Overview:
I have been running thousands of automated test cases as part of our nightly regression testing. One of the challenges in running this is analyzing the test results next day morning – Particularly when you have large set of regression test cases. Even though I have tried to keep the test flakiness as low as possible, there could be many genuine failures in the test results due to open defects. It is good that your automated regression suite works great by reporting them as issues. But you need to spend some time to find out if they are new issues or some existing issues in the issue tracking system.
To further simplify our work, We could add a feature in our test framework to automatically decide if the particular test should be executed or not. That is what we are going to see in this article.
Assumptions:
- I assume you use TestNG or similar framework.
- Your Issue tracking system has APIs enabled. For this article, I will be using GitHub as our issue tracker.
Sample testNG Test:
For better understanding, Lets create a sample testNG test first.
public class UserRegistrationTest {
@Test
public void launchRegistrationPage() {
RegistrationPage.launch();
Assert.assertTrue(RegistrationPage.isAt());
}
@Test(dependsOnMethods= "launchRegistrationPage")
public void enterDemographicInformation() {
RegistrationPage.enterDemographicInformation();
RegistrationPage.submit();
Assert.assertTrue(ProductsListPage.isAt());
}
@Test(dependsOnMethods= "enterDemographicInformation")
public void chooseProduct() {
ProductsListPage.chooseProduct(Products.DUMMY);
ProductsListPage.submit();
Assert.assertTrue(PaymentInformationPage.isAt());
}
@Test(dependsOnMethods= "chooseProduct")
public void enterPayment() {
PaymentInformationPage.enterPayment(PaymentMethods.CC);
PaymentInformationPage.submit();
Assert.assertTrue(OrderConfirmationPage.isAt());
}
@Test(dependsOnMethods= "enterPayment")
public void orderConfirmation() {
Assert.assertTrue(OrderConfirmationPage.isOrderSuccessful());
OrderConfirmationPage.close();
}
}In the above test, lets assume that we have been facing some issues with payment system – so all the tests are failing. As It is known issue, we do not want them to appear in the failed tests list as it increases the failed tests count and wastes our time. So , we would like to skip the test as long as the issue is still open.
IssueTrackerUtil:
In order to enhance our testNG framework, I create the following.
public enum IssueStatus {
OPEN,
CLOSED;
}
public class IssueTrackerUtil {
private static final String ISSUE_TRACKER_API_BASE_URL = "https://api.github.com/repos/username/dummy-project/issues/";
public static IssueStatus getStatus(String issueID) {
String githubIssueStatus = "CLOSED";
try {
githubIssueStatus = Unirest.get(ISSUE_TRACKER_API_BASE_URL.concat(issueID))
.asJson()
.getBody()
.getObject()
.getString("state")
.toUpperCase();
} catch (Exception e) {
e.printStackTrace();
}
return IssueStatus.valueOf(githubIssueStatus);
}
}I used Unirest library to make a call to the REST API. To include in your maven project,
<dependency>
<groupId>com.mashape.unirest</groupId>
<artifactId>unirest-java</artifactId>
<version>1.4.9</version>
</dependency>Issue Status Listener:
First, I would like to mention the open defects in the in testNG tests as shown here.
@Issue("APP-1234")
@Test(dependsOnMethods= "chooseProduct")
public void enterPayment() {
PaymentInformationPage.enterPayment(PaymentMethods.CC);
PaymentInformationPage.submit();
Assert.assertTrue(OrderConfirmationPage.isAt());
}So, I create an Issue annotation.
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface Issue {
String value() default "";
}Now we need to get the issue ID for each tests before it gets executed, call the above IssueTrackerUtil to get the issue status and skip the tests based on the status. Lets use testNG’s IInvokedMethodListener as shown here.
public class MethodIssueStatusListener implements IInvokedMethodListener {
@Override
public void afterInvocation(IInvokedMethod invokedMethod, ITestResult result) {
// nothing to do
}
@Override
public void beforeInvocation(IInvokedMethod invokedMethod, ITestResult result) {
Issue issue = invokedMethod.getTestMethod()
.getConstructorOrMethod()
.getMethod()
.getAnnotation(Issue.class);
if (null != issue) {
if (IssueStatus.OPEN.equals(IssueTrackerUtil.getStatus(issue.value()))) {
throw new SkipException("Skipping this due to Open Defect - " + issue.value());
}
}
}
}The rest is simple. Add the above listener in the testNG test class (usually in the base class or suite xml file) & mention the issue ID if any of the tests is associated with particular issue.
@Listener(MethodIssueStatusListener.class)
public class UserRegistrationTest {
@Test
public void launchRegistrationPage() {
RegistrationPage.launch();
Assert.assertTrue(RegistrationPage.isAt());
}
@Test(dependsOnMethods= "launchRegistrationPage")
public void enterDemographicInformation() {
RegistrationPage.enterDemographicInformation();
RegistrationPage.submit();
Assert.assertTrue(ProductsListPage.isAt());
}
@Test(dependsOnMethods= "enterDemographicInformation")
public void chooseProduct() {
ProductsListPage.chooseProduct(Products.DUMMY);
ProductsListPage.submit();
Assert.assertTrue(PaymentInformationPage.isAt());
}
@Issue("APP-1234")
@Test(dependsOnMethods= "chooseProduct")
public void enterPayment() {
PaymentInformationPage.enterPayment(PaymentMethods.CC);
PaymentInformationPage.submit();
Assert.assertTrue(OrderConfirmationPage.isAt());
}
@Test(dependsOnMethods= "enterPayment")
public void orderConfirmation() {
Assert.assertTrue(OrderConfirmationPage.isOrderSuccessful());
OrderConfirmationPage.close();
}
}When we execute the above test, I get the results as shown here.
Updating Issue Status Automatically:
Lets further enhance this!
Instead of skipping the tests, lets execute it every time.
- If it it fails, change the result to SKIPPED (Because we know that it might fail. We just ran to see if there is any luck!)
- If it passes, close the defect automatically.
public class IssueTrackerUtil {
private static final String ISSUE_TRACKER_API_BASE_URL = "https://api.github.com/repos/username/dummy-project/issues/";
private static final String ISSUE_TRACKER_USERNAME = "username";
private static final String ISSUE_TRACKER_PASSWORD = "password";
public static IssueStatus getStatus(String issueID) {
String githubIssueStatus = "CLOSED";
try {
githubIssueStatus = Unirest.get(ISSUE_TRACKER_API_BASE_URL.concat(issueID))
.asJson()
.getBody()
.getObject()
.getString("state")
.toUpperCase();
} catch (Exception e) {
e.printStackTrace();
}
return IssueStatus.valueOf(githubIssueStatus);
}
public static void updateStatus(String issueID, IssueStatus status) {
try {
Unirest.post(ISSUE_TRACKER_API_BASE_URL.concat(issueID).concat("/comments"))
.header("accept", ContentType.APPLICATION_JSON.toString())
.basicAuth(ISSUE_TRACKER_USERNAME, ISSUE_TRACKER_PASSWORD)
.body("{ \"body\" : \" testng method passed now. closing automatically.\" }")
.asJson();
Unirest.patch(ISSUE_TRACKER_API_BASE_URL.concat(issueID))
.header("accept", ContentType.APPLICATION_JSON.toString())
.basicAuth(ISSUE_TRACKER_USERNAME, ISSUE_TRACKER_PASSWORD)
.body("{ \"state\" : \""+ status.toString().toLowerCase() + "\" }")
.asJson();
} catch (UnirestException e) {
e.printStackTrace();
}
}
}
MethodIssueStatusListener is updated to take action after method invocation as shown here.
public class MethodIssueStatusListener implements IInvokedMethodListener {
@Override
public void afterInvocation(IInvokedMethod invokedMethod, ITestResult result) {
Issue issue = invokedMethod.getTestMethod()
.getConstructorOrMethod()
.getMethod()
.getAnnotation(Issue.class);
if (null != issue) {
if(IssueStatus.OPEN.equals(IssueTrackerUtil.getStatus(issue.value()))){
switch(result.getStatus()){
case ITestResult.FAILURE:
// no need to fail as we might have expected this already.
result.setStatus(ITestResult.SKIP);
break;
case ITestResult.SUCCESS:
// It is a good news. We should close this issue.
IssueTrackerUtil.updateStatus(issue.value(), IssueStatus.CLOSED);
break;
}
}
}
}
@Override
public void beforeInvocation(IInvokedMethod invokedMethod, ITestResult result) {
// nothing to do
}
}That’s it! When I execute my testNG tests, based on the tests status, it updates the issue status accordingly.
Summary:
By using GitHub API, testNG method listener, we are able to improve our existing framework to skip tests based on the status of the issue or update issue status based on the test results etc.
It saves our time by skipping the known issues . You could modify IssueTrackerUtil based on issue tracking system you use.
Happy Testing & Subscribe 🙂
