Goal:
To create a Continuous Delivery Pipeline which includes Performance Testing process to detect any performance related issues as early as possible.
Usually the full scale Performance Test will be done in the Staging/Pre-Production environment which could be identical to your Production environment. Code push to Staging happens after thorough QA functional/regression verification is done. So even if QA certifies the build, there is a chance that code might not work as expected at certain load which could be found only in Staging environment. This performance issue requires code change -> build -> QA verification once again which could postpone your Production push!
So, Can we fit the performance testing into the existing continuous development & testing process to detect the performance issues as early as possible?
Yes! We absolutely can! The idea here is to run the performance test in the lower environments [Dev/QA] (we might have to adjust the load accordingly) & to compare the results against some baseline metrics. Note that this is NOT going to replace the full scale performance test which we will do in the Staging environment.
Note: If you have not read this post – I would advise you to read the Part 1 of this post which talks about JMeter + Ant integration.
Jenkins Install:
Please check this link for the detailed steps on installing Jenkins on various OS.
Create Jenkins Job:
- Create a simple free style project.
- Build step should be ‘Invoke Ant‘
[We will use Default Ant assuming the slave machine has Ant installed already. if not, Let jenkins install ANT automatically.]
- We will invoke the target ‘all‘ as the target ‘all’ executes below tasks we wanted to execute.
- clean
- show-test-properties
- run
- generate-report
- generate-charts
- our test uses below properties. We will be passing the values for those properties from Jenkins to JMeter through Ant as given above.
- threadgroup.count
- threadgroup.rampup
- threadgroup.duration
- Our project folder is present under C:/workspace/CPT. So set the custom workspace accordingly.
- Our tests creates the results under result folder. We need to archive the files we need as given below. To archive all files under result, use result\*.*
Note: Jenkins will look for the files under the workspace. So set the path relative to the workspace.
Invoke JMeter Test from Jenkins:
- If all the above steps have been done correctly, Clicking on ‘Build’ will do to run the jmeter test. We can see a nice output as shown below. It runs the test for 3 users with 3 seconds as rampup period for 100 seconds.
- Once the test run is complete, Jenkins archives all the files under result folder. It includes the HTML file + PNG charts we have created. It archive the results for each and every run & they get stored in the server where jenkins is running.
- Instead of using hardcoded test properties, we can let our team member enter these values in Jenkins UI. To make it work, we need to make this job parameterized & Let’s create 3 parameters as our test expects 3 parameters to run. We can also have default values for those parameters.
- Modify the Build -> Properties section to use the Jenkins parameters we have created. This step is required because Ant expects values for these variables ‘threadgroup.count’..etc. [You can also avoid this step by creating the jenkins parameters like this : Instead of USER_COUNT, We can have create threadgroup.count as Jenkins Parameter name. ]. Since Jenkins parameters and Ant parameters are different , we should map as shown below.
- That’s it – Now, We should be able to pass the parameter’s values required for the JMeter test directly from Jenkins UI. Enter some values and click on Build. Console will show the new values passed to the test and JMeter will run the test accordingly. Below output shows the JMeter invokes 5 threads.
Jenkins-Performance Plugin:
Jenkins has a plugin for JMeter to parse the result files, create an aggregate report, create charts and to compare the current result with previous results etc.
You can find more details on the plugin here.
[Note that: This plugin parses the result file only if the results are stored in the XML format. If you prefer to use CSV format, It will not work]
Once the plugin is installed, for the Jenkins Job, we can find ‘Publish Performance test result report‘ under Post build actions. In the Report files section, give the relative path (to the workspace) of the output jtl file.
- When we run our tests couple of times, we can see Jenkins starts comparing the result of the performance test with previous test and shows the nice summary output. [You see a lot of 0 as response times in the below summary because i use a debug sampler for this demo which does not take time to execute. Response time chart shows a starighline because they are always 0 in my tests.]
- To get more details on each sampler, click on the sampler. Jenkins will provide the response times charts for each and every sampler as shown below. It also shows the data in the tabular format along with HTTP response codes which is nice!!.
EMailing the Results:
Jenkins has a nice plugin for Emailing the results. Please check this link for more details.
Once the plugin is installed, You can find a post build action ‘Editable Email Notification‘
- Under triggers of the Editable Email Notification, We will be setting the trigger as ‘Always’ [Always -> whether test fails or not. You can also change the trigger to send the mail only if the test passes etc]
- Once JMeter runs the test, the post build actions are getting executed consecutively in the order they are present in Jenkins Job/Project config. So, When this ‘post build action’ gets executed, the result files would have been already created under the result folder. We might want to include the HTML file in the Email body and attach the images. [If you do not want HTML in the Email body or you use TXT format, attach the HTML file as well as attachments along with images]
- Below screenshot will give an idea of the basic config to send the mail. [I assume you have already setup email server configuration or your Jenkins administrator will do that]
- Lets run our test one more time to see if Jenkins can send out the results for us after running the test.
- And..Yes….I got the email as shown here!!!
- All the charts files are also attached in the email received.
Summary:
We did a nice job by integrating JMeter with Ant and Jenkins. Thus Continuous Performance Testing process setup is implemented. We are also able to run the test as and when we want by a simple click in Jenkins. While you are concentrating on other tasks, Jenkins takes care of running the test, creating the results and sending out the results for you!! It also reduces the dependency on Performance Testers. YES..!! Anyone can run the test now. You have to just the share the link for the Jenkins job you have created.
We can also make this job integrated with ‘development deployments’ jobs of Jenkins – That is…whenever a code is pushed to the given test environment, this job gets executed automatically without any manual intervention. Any functional test and performance test can be done as early as possible to detect any issues upfront!!
Grafana Implementation for Real Time results:
The JMeter-Jenkins integration, emailing the results are great!! But If you had noticed, to get the results, we need to wait for the test to finish. Would it not be awesome to see the results while Jenkins is running the test!!??
If you have a long running test like mine and you are curious of seeing the current results while Jenkins is running the test – please see one of my favorite posts on getting real time results.
Happy Testing & Subscribe 🙂
