JMeter – Continuous Performance Testing – JMeter + ANT + Jenkins Integration – Part 1

38 Comments / Articles, Best Practices, CI / CD / DevOps, Framework, JMeter / By /

Goal:

To create a Continuous Delivery Pipeline which includes Performance Testing process to detect any performance related issues as early as possible.

 

continuous-performance-test

Usually the full scale Performance Test will be done in the Staging/Pre-Production environment which could be identical to your Production environment. Code push to Staging happens after thorough QA functional/regression verification is done. So even if QA certifies the build, there is a chance that code might not work as expected at certain load which could be found only in Staging environment. This performance issue requires code change -> build -> QA verification once again which could postpone your Production push!

So, Can we fit the performance testing into the existing continuous development & testing process to detect the performance issues as early as possible?

Yes! We absolutely can! The idea here is to run the performance test in the lower environments [Dev/QA] (we might have to adjust the load accordingly) & to compare the results against some baseline metrics. Note that this is NOT going to replace the full scale performance test which we will do in the Staging environment.

Lets see how we can implement the above process in this post by using JMeter + Ant + Jenkins.

Create Performance Test Script:

  1. Create some basic performance test script using JMeter.
  2. Check below useful articles from TestAutomationGuru.
    1. Creating Modular JMeter Test Scripts.
    2. Ensure that the script can be run on any given environment like QA/PPE/Staging etc. [Check here]
  3. I am going to consider a simple JMeter test which will login, search for a product and logout.
    1. I have used variables in the Thread Count, Ramp Up and Duration fields.
    2. At run time, we will pass values for those variables from Jenkins.
    3. To run the test in design mode/standalone mode, We use User Defined Variables as shown here. So, it takes the default values (like 1 user with 1 second ramp-up for 5 seconds test duration).
  4. Ensure that the scripts runs fine.

jmint-01                         jmint-02

 

Running JMeter test using ANT:

ANT: It is a build tool – here we will be using ANT to execute a set of tasks in the given order. For ex:

  • Cleaning the project by removing the temp files for a fresh test.
  • Running the test by passing the test properties from Jenkins to JMeter.
  • Creating reports
  • Creating charts

You would be calling these tasks once you are done with the build and unit/integration tests.

Install ANT:

  1. Download Ant from here.
  2. Uncompress the downloaded file into a directory.
  3. Set environmental variables
    • JAVA_HOME to your Java environment
    • ANT_HOME to the directory you uncompressed Ant
    • Add ${ANT_HOME}/bin (Unix) or %ANT_HOME%/bin (Windows) to your PATH.
    • If you do not have a JMETER_HOME variable, please set that too. It should point to the JMeter folder where you see ‘bin’
  4. In the cmd prompt/terminal, type ant and enter as given below. You should see below message (The system should know that there is an ‘ant’ command)

jmint-03

Create ANT-JMeter Project:

Now we are going to create an ANT project as given below.  [I have shared this sample project in GitHub]

jmint-04test folder is going to contain your .jmx file

lib folder will have all the libs required for ant-jmeter task + any other libs you want to include for your jmeter test.

function folder will have all the beanshell scripts for your test.

build.properties A property file which is going to pass the values for JMeter UDV.

jmint-05

ANT Build file:

This is an important part. Ant needs a build.xml file (it does not have to be ‘build.xml’ – but that is a default name ant will expect) where you have all the tasks defined.
Lets create a build.xml file as given below. [I have shared this sample project in GitHub]

jmint-06

We have 3 targets here.

  • clean -> cleaning the temp folders created as part of our test
  • show-test-properties -> Displays the values we pass to the test
  • run -> Runs the jmeter test

Now in the command prompt/terminal, go to the project folder and type ‘ant show-test-properties‘. You should see below output.

jmint-07

ANT: Run JMeter Test:

Lets run the test by issuing command:

ant run

jmint-08

Now we can see we have the log and result files created.

jmint-09

jmint-10

 

 

 

Creating HTML Report:

Run below command to generate the HTML report under output folder

ant generate-report

ant console log:

jmint-13

Report:

new-dashboard

 

 

Creating Charts:

 

The above HTML file itself has nice charts. However you still might want to create charts through command-line to send via emails/slack etc.

To create charts, we need few external plugins.

Run below command to install the required plugins.

ant install-plugins

Once plugins are installed, Generate charts by issuing command:

ant generate-charts

Once the job is complete, We can see all the charts we wanted are generated and placed under the result folder.

jmint-c-2

jmint-c-1

 

ANT – Running all tasks sequentially:

Run all ant tasks by issuing a command: ant all

jmint-16

 

Sample Project:

  • You can clone this sample project from here.
  • I am allocating 5GB memory for JMeter test. You can modify the value as you wish in the build.properties

Having Issues?

  • I see some comments from the readers that JMeter task is not executed by Ant. If possible, upgrade Java & Ant. [Some version of Java 8 & ant did not get along]
  • If it still does not work, execute the task below command to enable ant to provide us more information to debug.

ant -v run

 

Summary:

Now we will be able to run our JMeter test by passing the test properties from a property file through ANT. We also have automated creating HTML file and creating charts from the result file.

We will see how to invoke this test from Jenkins in the next post.

 

Thanks to [Ref: http://www.programmerplanet.org/projects/jmeter-ant-task/]

 

 

 

 

 

 

Share This:

38 thoughts on “JMeter – Continuous Performance Testing – JMeter + ANT + Jenkins Integration – Part 1”

  1. Hi,
    excellent article .

    I have downloaded your project and try to run but the jtl file was not creating under result folder.
    I ran ant generate-report it is showing jtl doesn’t exist.

    Reply

  2. Hi,
    This is a great article.
    I’d like to enrich report with PerfMon statistics such as CPU Utilization, Memory usage etc
    But, I’ve faced with issue how create xslt file for PerfMOn jtl files.
    They have completely different structure.
    Do you have any suggestions how to do it?

    Thanks a lot

    Reply

  3. Hi

    Can you please tell me the use of “ant-contrib-1.0b3.jar” that is included as part of this project. I could see one entry for this in build file as well, however, could not understand the functionality of it.

    Reply

  4. Great Post!! . But when the chart reports are getting generated printing exception “java.lang.Throwable: Could not access C:\Users\mypath\Documents\jmeter\lib\ext\lib
    at kg.apc.cmd.UniversalRunner.buildUpdatedClassPath(UniversalRunner.java:109)
    at kg.apc.cmd.UniversalRunner.(UniversalRunner.java:55)” . It is not impacting any flow I guess as I am getting build successful message. But could you please update what I am doing wrong here ? Thanks again.

    Reply

    1. Please ensure that ‘CMDRunner.jar’ is present in ‘%JMETER_HOME/lib/ext’ folder & you have latest version of Java.

      Reply

  5. Hi,

    I have downloaded your project and try to run but the jtl file was not creating under result folder.
    I ran ant generate-report it is showing jtl doesn’t exist.jave 1.8 and ant 1.1.1 is also there.
    Please share your thoughts

    Reply

    1. Some ant and java version (1.8) have compatibility issues – so it does not execute jmeter test at all. I have java 1.7.0.95 and ant 1.9.3

      Reply

    1. It is a link for download. Because of poor style, it does not show as a link. Just click on ‘CPT Project (683 downloads)’

      Reply

    1. You should not be adding any listeners to the test. Once the result file (.jtl) is generated, you can open the .jtl file in any listener you want.

      Reply

  8. “”We can see a nice HTML aggregate report as given below. (You can also modify the XSLT file to include throughput details).””

    How can this be modified just call the throughput object from the JTL i dont have exposure to xslt

    How can we add 90,95 percentiles to the HTML report ?

    Reply

  9. D:\apache-jmeter-2.13\extras\CPT>ant run
    Buildfile: D:\apache-jmeter-2.13\extras\CPT\build.xml

    clean:

    show-test-properties:
    [echo] JMETER_HOME : D:\apache-jmeter-2.13\bin
    [echo] TEST_PATH : D:\apache-jmeter-2.13\extras\CPT/test
    [echo] TEST_NAME : Test.123.jmx
    [echo] THREADGROUP_COUNT : 2
    [echo] THREADGROUP_RAMPUP : 2
    [echo] THREADGROUP_DURATION : 60

    run:
    [mkdir] Created dir: D:\apache-jmeter-2.13\extras\CPT\result
    [mkdir] Created dir: D:\apache-jmeter-2.13\extras\CPT\log

    BUILD FAILED
    D:\apache-jmeter-2.13\extras\CPT\build.xml:64: jmeter jar file not found or not a valid file: D:\apache-jmeter-2.13\bin\bin\ApacheJMeter.jar

    Reply

      1. Thanks a lot for this reply ,but after the changes I’m seeing the test is not getting triggered .

        D:\apache-jmeter-2.13\extras\CPT>ant run
        Buildfile: D:\apache-jmeter-2.13\extras\CPT\build.xml

        clean:
        [delete] Deleting directory D:\apache-jmeter-2.13\extras\CPT\result
        [delete] Deleting directory D:\apache-jmeter-2.13\extras\CPT\log

        show-test-properties:
        [echo] JMETER_HOME : D:\apache-jmeter-2.13\
        [echo] TEST_PATH : D:\apache-jmeter-2.13\extras\CPT/test
        [echo] TEST_NAME : AuditcreateCloudAPISigning1.7_3_Queue.jmx
        [echo] THREADGROUP_COUNT : 2
        [echo] THREADGROUP_RAMPUP : 2
        [echo] THREADGROUP_DURATION : 60

        run:
        [mkdir] Created dir: D:\apache-jmeter-2.13\extras\CPT\result
        [mkdir] Created dir: D:\apache-jmeter-2.13\extras\CPT\log
        [jmeter] Executing test plan: D:\apache-jmeter-2.13\extras\CPT\test\Test123.jmx ==> D:\apache-jmeter-2.13\extras\CPT\result\result.jtl
        [jmeter] Creating summariser
        [jmeter] Created the tree successfully using D:\apache-jmeter-2.13\extras\CPT\test\Test123.jmx
        [jmeter] Starting the test @ Mon Nov 21 20:18:56 IST 2016 (1479739736666)
        [jmeter] Waiting for possible shutdown message on port 4445
        [jmeter] summary = 0 in 0s = ******/s Avg: 0 Min: 9223372036854775807 Max: -9223372036854775808 Err: 0 (0.00%)
        [jmeter] Tidying up … @ Mon Nov 21 20:18:57 IST 2016 (1479739737227)
        [jmeter] … end of run

        BUILD SUCCESSFUL
        Total time: 2 seconds

        Reply

  10. Thanks vlns ,

    After the changes , Im seeing the test is not getting executed .

    D:\apache-jmeter-2.13\extras\CPT>ant run
    Buildfile: D:\apache-jmeter-2.13\extras\CPT\build.xml

    clean:
    [delete] Deleting directory D:\apache-jmeter-2.13\extras\CPT\result
    [delete] Deleting directory D:\apache-jmeter-2.13\extras\CPT\log

    show-test-properties:
    [echo] JMETER_HOME : D:\apache-jmeter-2.13\
    [echo] TEST_PATH : D:\apache-jmeter-2.13\extras\CPT/test
    [echo] TEST_NAME : Test123.jmx
    [echo] THREADGROUP_COUNT : 2
    [echo] THREADGROUP_RAMPUP : 2
    [echo] THREADGROUP_DURATION : 60

    run:
    [mkdir] Created dir: D:\apache-jmeter-2.13\extras\CPT\result
    [mkdir] Created dir: D:\apache-jmeter-2.13\extras\CPT\log
    [jmeter] Executing test plan: D:\apache-jmeter-2.13\extras\CPT\test\Test123.jmx ==> D:\apache-jmeter-2.13\extras\CPT\result\result.jtl
    [jmeter] Creating summariser
    [jmeter] Created the tree successfully using D:\apache-jmeter-2.13\extras\CPT\test\Test123.jmx
    [jmeter] Starting the test @ Mon Nov 21 20:18:56 IST 2016 (1479739736666)
    [jmeter] Waiting for possible shutdown message on port 4445
    [jmeter] summary = 0 in 0s = ******/s Avg: 0 Min: 9223372036854775807 Max: -9223372036854775808 Err: 0 (0.00%)
    [jmeter] Tidying up … @ Mon Nov 21 20:18:57 IST 2016 (1479739737227)
    [jmeter] … end of run

    BUILD SUCCESSFUL
    Total time: 2 seconds

    Reply

  11. Sorry , it was mistake from my end , Some tags I have mentioned incorrectly .

    This blog is a great blog . the way author mentioned and placed the steps . Its easy to execute a test from ant .

    Reply

  12. I really hope you can help, I have pulled down your CPT project, and run it, and it worked…but for some reason, now I have rebooted my pc, I can’t get it to work atall… when i run “ant run” i get –
    c:\CPT>ant run
    Buildfile: c:\CPT\build.xml

    clean:
    [delete] Deleting directory c:\CPT\result
    [delete] Deleting directory c:\CPT\log

    show-test-properties:
    [echo] JMETER_HOME : C:\apache-jmeter-3.0
    [echo] TEST_PATH : c:\CPT/test
    [echo] TEST_NAME : tag.jmx
    [echo] THREADGROUP_COUNT : 2
    [echo] THREADGROUP_RAMPUP : 2
    [echo] THREADGROUP_DURATION : 60

    run:
    [mkdir] Created dir: c:\CPT\result
    [mkdir] Created dir: c:\CPT\log
    [jmeter] Executing test plan: c:\CPT\test\tag.jmx ==> c:\CPT\result\result.jt
    l

    BUILD SUCCESSFUL
    Total time: 1 second

    And jmeter didn’t execute, like i said, this was all working but some reason has now stopped…any idea why?

    Thanks

    Reply

    2. Can you comment / remove below properties in the ant run task & try once?

      jmeter.save.saveservice.assertion_results
      jmeter.save.saveservice.thread_counts
      jmeter.save.saveservice.bytes
      jmeter.save.saveservice.response_data.on_error

      Reply

  13. I got it all working on Ubuntu, so im not going to bother with windows…so ignore my last comment, One thing I can’t get working is the new report, every time i run “ant generate-new-report” i get – “result.jtl’ does not contain the field names header, ensure the jmeter.save.saveservice.* properties are the same as when the CSV file was created or the file may be read incorrectly
    [java] An error occurred: Could not read sample
    [java] Java Result: 1″

    And I get that no matter what I try to run…any idea on why this isn’t working?

    Reply

    1. jmeter 3.0 expects the result file to be csv format. I updated the post now. The default one writes the result in xml file. Sorry for the confusion.

      Reply

  14. Greetings. It seems that jmeter v3.0 report creation requires
    , does it?

    Because if I leave “xml” value, I’m getting

    generate-new-report:
    [mkdir] Created dir: C:\PerfTest\CI\output
    [java] Writing log file to: C:\PerfTest\CI\jmeter.log
    [java] File ‘C:\PerfTest\CI\results\result.jtl’ does not contain the field names header, ensure the jmeter.save.saveser
    vice.* properties are the same as when the CSV file was created or the file may be read incorrectly
    [java] An error occurred: Could not read sample
    [java] Java Result: 1

    Reply

  15. result shows NAN….here’s the ant -v run output:

    C:\workspace\CPT>ant -v run
    Apache Ant(TM) version 1.8.4 compiled on May 22 2012
    Trying the default build file: build.xml
    Buildfile: C:\workspace\CPT\build.xml
    Detected Java version: 1.7 in: C:\Program Files\Java\jdk1.7.0_67\jre
    Detected OS: Windows 7
    parsing buildfile C:\workspace\CPT\build.xml with URI = file:/C:/workspace/CPT/b
    uild.xml
    Project base dir set to: C:\workspace\CPT
    parsing buildfile jar:file:/C:/Program%20Files/apache-ant-1.8.4-bin/apache-ant-1
    .8.4/lib/ant.jar!/org/apache/tools/ant/antlib.xml with URI = jar:file:/C:/Progra
    m%20Files/apache-ant-1.8.4-bin/apache-ant-1.8.4/lib/ant.jar!/org/apache/tools/an
    t/antlib.xml from a zip file
    [property] Loading C:\workspace\CPT\build.properties
    [property] Loading Environment env.
    [pathconvert] Set property jmeter.classpath = C:\workspace\CPT\lib\CMDRunner.jar
    :C:\workspace\CPT\lib\ant-contrib-1.0b3.jar:C:\workspace\CPT\lib\ant-jmeter-1.1.
    1.jar:C:\workspace\CPT\lib\serializer-2.7.1.jar:C:\workspace\CPT\lib\xalan-2.7.1
    .jar
    parsing buildfile jar:file:/C:/workspace/CPT/lib/ant-contrib-1.0b3.jar!/net/sf/a
    ntcontrib/antlib.xml with URI = jar:file:/C:/workspace/CPT/lib/ant-contrib-1.0b3
    .jar!/net/sf/antcontrib/antlib.xml from a zip file
    Build sequence for target(s) `run’ is [clean, show-test-properties, run]
    Complete build sequence is [clean, show-test-properties, run, generate-report, g
    enerate-chart, all, ]

    clean:

    show-test-properties:
    [echo] JMETER_HOME : C:\apache-jmeter-2.8
    [echo] TEST_PATH : C:\workspace\CPT/test
    [echo] TEST_NAME : fwaesinsgateway.jmx
    [echo] THREADGROUP_COUNT : 10
    [echo] THREADGROUP_RAMPUP : 5
    [echo] THREADGROUP_DURATION : 60

    run:
    [mkdir] Created dir: C:\workspace\CPT\result
    [mkdir] Created dir: C:\workspace\CPT\log
    [jmeter] Using JMeter Home: C:\apache-jmeter-2.8
    [jmeter] Using JMeter Jar: C:\apache-jmeter-2.8\bin\ApacheJMeter.jar
    [jmeter] Executing test plan: C:\workspace\CPT\test\fwaesinsgateway.jmx ==> C
    :\workspace\CPT\result\result.jtl
    [jmeter] Executing ‘C:\Program Files\Java\jdk1.7.0_67\jre\bin\java.exe’ with
    arguments:
    [jmeter] ‘-Xmx5120m’
    [jmeter] ‘-jar’
    [jmeter] ‘C:\apache-jmeter-2.8\bin\ApacheJMeter.jar’
    [jmeter] ‘-n’
    [jmeter] ‘-j’
    [jmeter] ‘C:\workspace\CPT\log\jmeter.log’
    [jmeter] ‘-t’
    [jmeter] ‘C:\workspace\CPT\test\fwaesinsgateway.jmx’
    [jmeter] ‘-l’
    [jmeter] ‘C:\workspace\CPT\result\result.jtl’
    [jmeter] ‘-Jjmeter.save.saveservice.output_format=xml’
    [jmeter] ‘-Jjmeter.save.saveservice.assertion_results=all’
    [jmeter] ‘-Jjmeter.save.saveservice.thread_counts=true’
    [jmeter] ‘-Jjmeter.save.saveservice.bytes=true’
    [jmeter] ‘-Jjmeter.save.saveservice.response_data.on_error=true’
    [jmeter] ‘-Jthreadgroup.count=10’
    [jmeter] ‘-Jthreadgroup.rampup=5’
    [jmeter] ‘-Jthreadgroup.duration=60’
    [jmeter]
    [jmeter] The ‘ characters around the executable and arguments are
    [jmeter] not part of the command.
    [jmeter] Created the tree successfully using C:\workspace\CPT\test\fwaesinsga
    teway.jmx
    [jmeter] Starting the test @ Fri Dec 23 00:52:22 SGT 2016 (1482425542546)
    [jmeter] Waiting for possible shutdown message on port 4445
    [jmeter] Tidying up … @ Fri Dec 23 00:52:22 SGT 2016 (1482425542656)
    [jmeter] … end of run

    BUILD SUCCESSFUL
    Total time: 1 second

    C:\workspace\CPT>ant generate-report

    Reply

  16. I try the first step after making the build.xml and get the below error:
    C:\workspace\CPT>ant show-test-properties
    Buildfile: C:\workspace\CPT\build.xml

    BUILD FAILED
    C:\workspace\CPT\build.xml:2: Unexpected element “{}target” {antlib:org.apache.t
    ools.ant}target

    Total time: 0 seconds

    C:\workspace\CPT>

    Reply

    1. Looks like build.xml is corrupted! Did you download once again and try? Please ensure that you have the latest version of Java and Ant

      Reply

  18. Hi,vlns
    I want to pass the values of ${__property(url,,)} in build.xml,but failed,would you like to tell me how to get it
    thanks

    Reply

  19. thanks vlns
    I tried with ,but it did not work
    passed with in build.xml,it works
    but I’m not sure if it’s a correct way.

    Reply

  20. Great info.
    I am having problem to run some of the commands.
    ant install-plugins / ant generate-charts – I am getting “Target “install-plugins” does not exist in the project “ant-jmeter”.
    Any suggestions?

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.