Goal:
To create a Continuous Delivery Pipeline which includes Performance Testing process to detect any performance related issues as early as possible.
Usually the full scale Performance Test will be done in the Staging/Pre-Production environment which could be identical to your Production environment. Code push to Staging happens after thorough QA functional/regression verification is done. So even if QA certifies the build, there is a chance that code might not work as expected at certain load which could be found only in Staging environment. This performance issue requires code change -> build -> QA verification once again which could postpone your Production push!
So, Can we fit the performance testing into the existing continuous development & testing process to detect the performance issues as early as possible?
Yes! We absolutely can! The idea here is to run the performance test in the lower environments [Dev/QA] (we might have to adjust the load accordingly) & to compare the results against some baseline metrics. Note that this is NOT going to replace the full scale performance test which we will do in the Staging environment.
Note: If you have not read this post – I would advise you to read the Part 1 of this post which talks about JMeter + Ant integration.
Jenkins Install:
Please check this link for the detailed steps on installing Jenkins on various OS.
Create Jenkins Job:
- Create a simple free style project.
- Build step should be ‘Invoke Ant‘
[We will use Default Ant assuming the slave machine has Ant installed already. if not, Let jenkins install ANT automatically.]
- We will invoke the target ‘all‘ as the target ‘all’ executes below tasks we wanted to execute.
- clean
- show-test-properties
- run
- generate-report
- generate-charts
- our test uses below properties. We will be passing the values for those properties from Jenkins to JMeter through Ant as given above.
- threadgroup.count
- threadgroup.rampup
- threadgroup.duration
- Our project folder is present under C:/workspace/CPT. So set the custom workspace accordingly.
- Our tests creates the results under result folder. We need to archive the files we need as given below. To archive all files under result, use result\*.*
Note: Jenkins will look for the files under the workspace. So set the path relative to the workspace.
Invoke JMeter Test from Jenkins:
- If all the above steps have been done correctly, Clicking on ‘Build’ will do to run the jmeter test. We can see a nice output as shown below. It runs the test for 3 users with 3 seconds as rampup period for 100 seconds.
- Once the test run is complete, Jenkins archives all the files under result folder. It includes the HTML file + PNG charts we have created. It archive the results for each and every run & they get stored in the server where jenkins is running.
- Instead of using hardcoded test properties, we can let our team member enter these values in Jenkins UI. To make it work, we need to make this job parameterized & Let’s create 3 parameters as our test expects 3 parameters to run. We can also have default values for those parameters.
- Modify the Build -> Properties section to use the Jenkins parameters we have created. This step is required because Ant expects values for these variables ‘threadgroup.count’..etc. [You can also avoid this step by creating the jenkins parameters like this : Instead of USER_COUNT, We can have create threadgroup.count as Jenkins Parameter name. ]. Since Jenkins parameters and Ant parameters are different , we should map as shown below.
- That’s it – Now, We should be able to pass the parameter’s values required for the JMeter test directly from Jenkins UI. Enter some values and click on Build. Console will show the new values passed to the test and JMeter will run the test accordingly. Below output shows the JMeter invokes 5 threads.
Jenkins-Performance Plugin:
Jenkins has a plugin for JMeter to parse the result files, create an aggregate report, create charts and to compare the current result with previous results etc.
You can find more details on the plugin here.
[Note that: This plugin parses the result file only if the results are stored in the XML format. If you prefer to use CSV format, It will not work]
Once the plugin is installed, for the Jenkins Job, we can find ‘Publish Performance test result report‘ under Post build actions. In the Report files section, give the relative path (to the workspace) of the output jtl file.
- When we run our tests couple of times, we can see Jenkins starts comparing the result of the performance test with previous test and shows the nice summary output. [You see a lot of 0 as response times in the below summary because i use a debug sampler for this demo which does not take time to execute. Response time chart shows a starighline because they are always 0 in my tests.]
- To get more details on each sampler, click on the sampler. Jenkins will provide the response times charts for each and every sampler as shown below. It also shows the data in the tabular format along with HTTP response codes which is nice!!.
EMailing the Results:
Jenkins has a nice plugin for Emailing the results. Please check this link for more details.
Once the plugin is installed, You can find a post build action ‘Editable Email Notification‘
- Under triggers of the Editable Email Notification, We will be setting the trigger as ‘Always’ [Always -> whether test fails or not. You can also change the trigger to send the mail only if the test passes etc]
- Once JMeter runs the test, the post build actions are getting executed consecutively in the order they are present in Jenkins Job/Project config. So, When this ‘post build action’ gets executed, the result files would have been already created under the result folder. We might want to include the HTML file in the Email body and attach the images. [If you do not want HTML in the Email body or you use TXT format, attach the HTML file as well as attachments along with images]
- Below screenshot will give an idea of the basic config to send the mail. [I assume you have already setup email server configuration or your Jenkins administrator will do that]
- Lets run our test one more time to see if Jenkins can send out the results for us after running the test.
- And..Yes….I got the email as shown here!!!
- All the charts files are also attached in the email received.
Summary:
We did a nice job by integrating JMeter with Ant and Jenkins. Thus Continuous Performance Testing process setup is implemented. We are also able to run the test as and when we want by a simple click in Jenkins. While you are concentrating on other tasks, Jenkins takes care of running the test, creating the results and sending out the results for you!! It also reduces the dependency on Performance Testers. YES..!! Anyone can run the test now. You have to just the share the link for the Jenkins job you have created.
We can also make this job integrated with ‘development deployments’ jobs of Jenkins – That is…whenever a code is pushed to the given test environment, this job gets executed automatically without any manual intervention. Any functional test and performance test can be done as early as possible to detect any issues upfront!!
Grafana Implementation for Real Time results:
The JMeter-Jenkins integration, emailing the results are great!! But If you had noticed, to get the results, we need to wait for the test to finish. Would it not be awesome to see the results while Jenkins is running the test!!??
If you have a long running test like mine and you are curious of seeing the current results while Jenkins is running the test – please see one of my favorite posts on getting real time results.
Happy Testing & Subscribe 🙂
Great Post !!
May I know if we can integrate this with the JMeter plugins thread group. So that I can pass the parameters like RampUp, Ramp Down, Hold For ..etc
And also you mentioned that we can get details like throughput 90th %ile in the HTML report generated by ANT task I tried but failed, it will be great if you can eloborate the above two points
Thanks in advance.
Thanks.
Yes, You can use the jmeter-plugins thread groups as well. Ensure that you parameterize correctly and place the jars in the slave machine as well.
Regarding the HTML report, you can update the XSLT file to add any additional information you want. But when i mean by ‘update’, you need to include the logic to calculate the information yourself. So, it might not be very easy in some cases.
If I find time, i will try to do that as a separate post.
Thanks, that would be useful
Do you have issues with Ant failing the Jmeter build, with no report?
I have, and expect it’s a java configuration, but would like to know more about how you configured your java/ant/maven heap settings.
Do you mean Ant does not execute JMeter task? I saw in StackOverFlow that there is some compatibility issue with Ant & Java versions. Please try to use latest versions and see If it fixes the issue you are facing.
Excellent and good one .
I am wondering what extra i would need to configure if i want to run more than one script .
Do i need to add all the jmeter script under the test ? IF yes how does ant understand which jmeter would run if i want to trigger some specifci script only .
Use testplans as shown here instead of test plan.
http://www.ibm.com/developerworks/library/j-ap04088/index.htm
Hello, Thanks for the very informative articles.
I was wonder how much value has performance testing CI provided for you in practice? Did you catch many significant performance issues early on?
Glad that you found this informative.
Yes, This process helped us from injecting 28 performance issues in PROD this year.
That sounds good!
Did you do full scale performance testing in CI or slimmed down test with limited user and limited scripts?
How much maintenance work on updating JMeter scripts what required as changes were made to the system?
We get frequent builds. Even before doing the full scale perf test we do an initial test with limited users & compare with the baseline. Mostly these are all APIs – maintenance is very easy. Even for functional testing (not for all functionalities of the application – focus is to test the server side logic), we use JMeter. These perf test scripts are sub set of those regression scripts.
Thank you!
I see a lot of article online regarding the technicalities of how to implement continuous performance test but not much on how successful its been on projects in various case studies.
Found the issue – I was setting same variables both from Jenkins and build.properties.
After I’ve removed build.properties and set up all variables in Jenkins only it works fine.
Thanks once again for your contribution – you are giving amazing tutorials and out-of-box solutions.
Glad that it worked.
Hi,
We have designed UI tests using Selenium webdriver in JMeter. We wanted to run these UI tests in Jenkins without using headless approach. Do we have any references or steps to run JMeter UI tests in Jenkins?
Is your aim to do functional testing using JMeter + WebDriver? If the test is working fine in your local and all the dependencies are present in remote machine(like java, webdriver, jmeter ect), there should not be any issues in running using Jenkins. you do not need Ant/Maven. Just this command (windows batch command/shell script in jenkins) – “jmeter -n -t test.jmx” will do. Do you face any specific issue in running the test using Jenkins?
Hi,vlus
while executing generate-chart target on jenkins,there is exception throwing,It would be great if you have time to help me figure out why
] Exception in thread “main” java.lang.UnsupportedOperationException: Seems you’ve requested aggregate mode for graph that don’t support it. We apologize…
[java] at kg.apc.jmeter.graphs.AbstractGraphPanelVisualizer.switchModel(AbstractGraphPanelVisualizer.java:363)
Yan, All the reports can not be generated in the aggregate mode. So, The exception is normal for certain reports. If you want to disable it, you can remove the “–aggregate-rows” and the next “yes” for the “generate-graph” target in the build.xml file.
Use this latest sample project – https://github.com/vinsguru/jmeter-ant
vlns,Thank you very much for your quick reply,issue has been fixed after remove the “–aggregate-rows” and the next “yes”
the latest project looks more excellent ,thanks again for your contribution
Hi,vlns
I use testplans to run several scripts once time in build.xml,but there is a problem for me,is it possible to generate jmeter results for all scripts into one test.jtl
Looking forward to your reply
Thanks
Check this JMeter plugin – https://jmeter-plugins.org/wiki/MergeResults/. It merges the results. If you want 1 combined result file, do not added any prefix label. If you are merging the results to compare the results, then you would need the prefix label.
This is an excellent post! I have question though, how is it different from running manual Jmeter GUI mode? Does it run on Jenkins server or the actual application servers?
Thanks!
By using Jenkins, you can configure the performance testing job as part of your CD pipeline. Jenkins will run the JMeter performance test using one of the slaves machines it has been configured to run. Not on the actual appn server.
I Have Already set Environment Variables But Geetin Errors
Started by user nisar
Building in workspace C:\Users\nisarahs\Documents\jmeter-ant-master
[jmeter-ant-master] $ cmd.exe /C “ant.bat -Dthreadgroup.duration=180 -Dthreadgroup.rampup=3 -Dthreadgroup.count=3 all && exit %%ERRORLEVEL%%”
Buildfile: C:\Users\nisarahs\Documents\jmeter-ant-master\build.xml
clean:
[delete] Deleting directory C:\Users\nisarahs\Documents\jmeter-ant-master\result
[delete] Deleting directory C:\Users\nisarahs\Documents\jmeter-ant-master\log
[delete] Deleting directory C:\Users\nisarahs\Documents\jmeter-ant-master\output
show-test-properties:
[echo] JMETER_HOME : C:\Users\nisarahs\Downloads\apache-jmeter-4.0
[echo] TEST_PATH : C:\Users\nisarahs\Documents\jmeter-ant-master/test/Apsrtc.jmx
[echo] THREADGROUP_COUNT : 3
[echo] THREADGROUP_RAMPUP : 3
[echo] THREADGROUP_DURATION : 180
run:
[mkdir] Created dir: C:\Users\nisarahs\Documents\jmeter-ant-master\result
[mkdir] Created dir: C:\Users\nisarahs\Documents\jmeter-ant-master\log
[jmeter] Executing test plan: C:\Users\nisarahs\Documents\jmeter-ant-master\test\Apsrtc.jmx ==> C:\Users\nisarahs\Documents\jmeter-ant-master\result\result.jtl
generate-report:
[mkdir] Created dir: C:\Users\nisarahs\Documents\jmeter-ant-master\output
[java] An error occurred: Cannot read test results file : C:\Users\nisarahs\Documents\jmeter-ant-master\result\result.jtl
[java] Java Result: 1
generate-chart:
[exec] ‘C:\Users\nisarahs\Downloads\apache-jmeter-4.0/bin/JMeterPluginsCMD’ is not recognized as an internal or external command,
[exec] operable program or batch file.
[exec] Result: 1
[exec] ‘C:\Users\nisarahs\Downloads\apache-jmeter-4.0/bin/JMeterPluginsCMD’ is not recognized as an internal or external command,
[exec] operable program or batch file.
[exec] Result: 1
[exec] ‘C:\Users\nisarahs\Downloads\apache-jmeter-4.0/bin/JMeterPluginsCMD’ is not recognized as an internal or external command,
[exec] operable program or batch file.
[exec] Result: 1
[exec] ‘C:\Users\nisarahs\Downloads\apache-jmeter-4.0/bin/JMeterPluginsCMD’ is not recognized as an internal or external command,
[exec] operable program or batch file.
[exec] Result: 1
[exec] ‘C:\Users\nisarahs\Downloads\apache-jmeter-4.0/bin/JMeterPluginsCMD’ is not recognized as an internal or external command,
[exec] operable program or batch file.
[exec] Result: 1
[exec] ‘C:\Users\nisarahs\Downloads\apache-jmeter-4.0/bin/JMeterPluginsCMD’ is not recognized as an internal or external command,
[exec] operable program or batch file.
[exec] Result: 1
[exec] ‘C:\Users\nisarahs\Downloads\apache-jmeter-4.0/bin/JMeterPluginsCMD’ is not recognized as an internal or external command,
[exec] operable program or batch file.
[exec] Result: 1
all:
BUILD SUCCESSFUL
Total time: 1 second
Archiving artifacts
ERROR: No artifacts found that match the file pattern “log*.log,result*.“. Configuration error?
ERROR: ‘log*.log’ doesn’t match anything, but ‘.log’ does. Perhaps that’s what you mean?
Build step ‘Archive the artifacts’ changed build result to FAILURE
Finished: FAILURE
did we need to keep ant and ant master inside jenkins Folder ?
Hi Nisar,
You seem to be using the latest version JMeter 4.0. The articles was created for JMeter version 3.0. Soemthing might have been changed. I need to take a look.
Hi,
It was a very nice article Im trying to implement this one challenge im facing Im using load genearators (Master slave concept) how can I leverage this issue with using jenkins ?
The concept is still same. You always connect the master to jenkins. Not slaves. Master is responsible for connecting to slaves and running the test.