In this post, I would like to show how I have implemented automated continuous regression testing process in my project. It is based on the Hybrid framework, I had implemented. I would request you to read the post on Hybrid Framework first if you have not.
Problem Statement:
- We follow agile methodology & we have a release once in every 4 weeks. For every release, we push tons of new features into the existing application.As part of the addition of new features/requirements, we also introduce new defects in the existing application.
- When we have 2 weeks for development, 1 week for QA & 1 week for deployment activities for a release, It is very hard to complete all the testing on all the new requirements & the regression testing to ensure that existing requirements of the application are still working as expected.
- New code is pushed to the QA environment daily! So, even if some test cases pass today, they might fail the next day as code is frequently modified!!!
- No one wants defect to be found in the last min as it is very costly to fix & it will also affect the release plan and subsequent releases. Detect defects as soon as they are introduced!!
Continuous Integration :
While developing a software, developers might check-in their code very often a day (at least once). This integration should be verified (using build tools, unit & integration tests) to ensure that software is not broken/developers do not introduce any defect. This process is called Continuous Integration & It is development practice to detect these code integration related issues as early as possible in the software development life cycle.
Above process is helpful in finding all the unit testing / integration testing related issues earlier. It might not find all the functional issues. Why do we, automation engineers, not follow the similar process to detect application functional defects earlier?
Automated Continuous Regression:
I have 5000 test cases in our automation regression suite which, an automated testing tool ,will take 90 hours to execute all the test cases as most of the test cases are very complex!!! All the 5000 test cases are split into multiple suite files based on the business functionality as given below in the picture 1.
We create a separate jenkins job for each suite file.(We use QTP for a project & Selenium-WebDriver for other. We use hybrid framework for both. So this process is not specific to QTP projects. It can be used for any automation testing project).
Each Jenkins job is scheduled to run daily @ 7PM by then we would have received the latest build from the development for the day with all the defect fixes. We have 10 slave machines connected to the jenkins master. So the entire automated regression suites are run daily by the jenkins master on different slave machines. By next day morning, All the 5000 test cases would have been executed & results would be ready ready for analysis.
Advantages:
- Huge effort saving by automating 5000 test cases.
- 0 hr spent in regression testing + few hours for result analysis.
- Immediate feedback – Any new defects introduced today will be caught by next day morning.
- Reduced time to market
- Improved the code quality.
- It has greatly reduced the unnecessary dependency on the QA team for the Regression status.
- Testing team focuses more on testing the new features planned for the release. Does not have to really worry about the regression testing.
Overall Application Regression Health Check:
I have NodeJS server running which serves this dashboard by querying the data from MongoDB which shows overall application functionality health check status. I have a multiple suites / jenkins jobs. Without this dashboard, Checking individual jobs/emails for my application status was very difficult.
Dashboard:
We also have created below matrix to display all the test case execution results for the past few weeks. This is helpful in triaging failed test cases. To know the build version in which a functionality was broken. So that developer can look into the code change happened on that date and easily fix the issue.
Summary:
Implementing this process is not very difficult for your organization. This post only shows a very high level idea about the process. If you are interested in implementing something similar for your organization, Please refer to below posts of TestAutomationGuru.
- QTP/UFT – Jenkins Integration
- Passing Parameters to UFT test from Jenkins
- Github / SVN integration for UFT scripts
Happy Testing 🙂
