Ocular – Automated Visual Validation for Selenium WebDriver Test Automation Frameworks

Overview:

Visual Validation is a process of validating the visual aspects of an application’s user interface – Verifying each element in the application is appearing exactly in the same size, color, shape etc.

Lets say, your application has charts to represent some data to the end user, how can we verify if the chart is appearing as expected?

Verifying the visual aspects using automated scripts is tedious. We might have to write tons of assertions! Mostly, automation engineers find this difficult to automate & take care of the application’s functionality testing and leave the UI related testing to the manual testers.

Even if the application layout changes, if the functionality works fine, automated test scripts never catch those issues!  Enhancing the automated tests to include the automated visual validation feature for an application is also very challenging!

 

Challenges in Visual Validation:

• Commercial visual validation tools are very expensive – (We were given a quote of $100K per year by one of the famous vendors – for one of the applications)
• Organizing / maintaining the baseline images for comparison for each pages / components of the application
• Responsive Testing – Same page appears in different layouts in different devices
• Reporting the differences
• Applications tend to change over time – Visual validation tests are always failing!
• Modifying existing framework is very difficult to include the visual validation feature.

 

TestAutomationGuru has come up with its own library, Ocular, for your selenium WebDriver test automation frameworks which could solve most of these challenges. It is FREE & Open Source!

Github:

Check here for the source.

---

Ocular:

Ocular is a simple java utility which helps us to add the visual validation feature into the existing WebDriver test automation frameworks.

(Ocular is developed on top of Arquillian RushEye which is a command-line image-comparison tool. Special thanks to Mr. Lukas who created the rusheye project & Arquillian’s team for maintaining it.

If you are an arquillian-graphene user, I am working on a Graphene Extension to include this ocular. I will update here when it is ready to use.)

Maven Dependency:

Place below artifact configuration into Maven dependencies just below your selenium-java dependency:

<dependency>
    <groupId>com.testautomationguru.ocular</groupId>
    <artifactId>ocular</artifactId>
    <version>1.0.0.Alpha</version>
</dependency>

Terms:

Ocular uses below terms.

• Snapshot – the baseline image.
• Sample – the actual image.
• Similarity – how similar the sample should be while comparing against snapshot.

---

Global Configuration:

Ocular expects the user to configure the following properties before doing the visual validation.

Configuration Property	Description	Type	Default Value
snapshotPath	location of the baseline images	Path	null
resultPath	location where the results with differences highlighted should be stored	Path	null
globalSimilarity	% of pixels should match for the visual validation to pass	int	100
saveSnapshot	flag to save the snapshots automatically if they are not present. (useful for the very first time test run)	boolean	true

Code example:

Update Ocular global configuration as shown here once – for ex: in your @BeforeSuite method.

Ocular.config()
    .resultPath(Paths.get("c:/ocular/result"))
    .snapshotPath(Paths.get("c:/ocular/snpshot"))
    .globalSimilarity(95)
    .saveSnapshot(false);

Note: Ocular does NOT create these directories. Please ensure that they are present.

---

Snapshot in Page Objects:

Ocular.config()
    .resultPath(Paths.get("c:/ocular/result"))
    .snapshotPath(Paths.get("c:/ocular/snpshot"))
    .globalSimilarity(99)
    .saveSnapshot(true);

A Page Object is created for the below page as shown here.

 

@Snap("RichFace.png")
public class RichFace {

    private final WebDriver driver;

    public RichFace(WebDriver driver){
        this.driver = driver;
    }

}

Now Ocular looks for the snapshot image at this location – c:/ocular/snpshot/RichFace.png – (snapshotPath + Path given in @Snap)- in order to do the visual validation.

---

 Note: If the image is not present in the location, Ocular creates image and place it under snapshot path. So that Ocular can use this image from the next run onward for comparison.

---

Comparing Snapshot against Sample:

Sample is the actual image. Sample is created by taking the screenshot of the current page using the WebDriver.

@Snap("RichFace.png")
public class RichFace {

    private final WebDriver driver;

    public RichFace(WebDriver driver) {
        this.driver = driver;
    }

    public OcularResult compare() {
        return Ocular.snapshot().from(this)     (1)
                     .sample().using(driver)    (2)
                     .compare();                (3)
    }
}

1. snpshot.from(this) – lets Ocular read the @Snap value
2. sample.using(driver) – lets Ocular to take the current page screenshot
3. compare() – compares the snapshot against sample and returns the result

---

Different ways to access Snapshot:

Ocular.snapshot()
    .from(this)     // or
    .from(RichFace.class) // or
    .from(Paths.get("/path/of/the/image"))

---

Choosing Right Snapshots at Run Time:

@Snap("RichFace-#{THEME}.png")
public class RichFace {

    private final WebDriver driver;

    public RichFace(WebDriver driver){
        this.driver = driver;
    }

    public OcularResult compare() {
        return Ocular
            .snapshot()
                .from(this)
                .replaceAttribute("THEME","ruby") // lets the ocular look for RichFace-ruby.png
            .sample()
                .using(driver)
            .compare();
    }
}

You could use some variables in the @Snap , get them replaced with correct values.

---

Excluding Elements:

Sometimes your application might have an element which could contain some non-deterministic values. For example, some random number like order conformation number, date/time or 3rd party ads etc. If we do visual validation, We know for sure that snapshot and sample will not match. So, We need to exclude those elements. It can be achieved as shown here.

Ocular.snapshot()
        .from(this)
    .sample()
        .using(driver)
        .exclude(element)
    .compare();

If we need to exclude a list of elements,

List<WebElement> elements = getElementsToBeExcluded();

Ocular.snapshot()
        .from(this)
    .sample()
        .using(driver)
        .exclude(elements)
    .compare();

or

Ocular.snapshot()
        .from(this)
    .sample()
        .using(driver)
        .exclude(element1)
        .exclude(element2)
        .exclude(element3)
    .compare();

---

Comparing One Specific Element:

Ocular can also compare a specific element instead of a whole web page.

Ocular.snapshot()
        .from(this)
    .sample()
        .using(driver)
        .element(element)
    .compare();

---

Similarity:

Sometimes we might not be interested in 100% match. Applications might change a little over time. So we might not want very sensitive compare. In this case, We could use similarity to define the % of pixels match. For the below example, If 85% of the pixels match, then Ocular will pass the visual validation. This will override the Ocular.config().globalSimilarity() config settings for this page object / fragment.

@Snap(value="RichFace-#{THEME}.png",similarity=85)
public class RichFace {

}

You could also override global similarity config value as shown here.

@Snap("RichFace-#{THEME}.png")
public class RichFace {

    private final WebDriver driver;

    public RichFace(WebDriver driver){
        this.driver = driver;
    }

    public OcularResult compare() {
        return Ocular.snapshot()
            .from(this)
            .replaceAttribute("THEME","ruby") // lets the ocular look for RichFace-ruby.png
        .sample()
            .using(driver)
            .similarity(85)
        .compare();
    }
}

---

Ocular Result:

OcularResult extends ComparisonResult class of arquillian-rusheye which has a set of methods & provides result of the visual validation. The image with differences highlighted is getting saved at Ocular.config().resultPath() every time!

OcularResult result = Ocular.snapshot().from(this)
             .sample().using(driver)
             .compare();

result.isEqualImages() // true or false
      .getSimilairty() // % of pixels match
      .getDiffImage()  // BufferedImage differences highlighted
      .getEqualPixels() // number of pixels matched
      .getTotalPixels() // total number of pixels

---

Sample Diff Image:

Ocular creates below diff image in case of snapshot & sample comparison mismatch highlighting the differences between them.

---

Responsive Web Design Testing:

Ocular can also be used to test the responsive web design of your application.

To the responsive web design, Lets organize our snapshots in our test framework as shown here.

src
	test
		resources
			ocular
				iphone
					PageA.png
					PageB.png
					PageC.png
				ipad
					PageA.png
					PageB.png
					PageC.png
				galaxy-tab
					PageA.png
					PageB.png
					PageC.png					
				1280x1024
					PageA.png
					PageB.png
					PageC.png					

Lets update the Ocular global config with the path where Ocular should expect the snapshots for the visual validation.

String device = "ipad";
Path path = Paths.get(".", "src/test/resources/ocular", device);
Ocular.config().snapshotPath(path);


// now Ocular will use the PageB.png under ipad folder.                 
@Snap(value="PageB.png")
public class PageB {
 
}

Now as shown above we could easily incorporate the responsive web design validation in our test framework.

---

Summary:

Ocular, with its static and fluent API, makes its easy for us to use the library & include the automated UI / Responsive web design testing feature in our test frameworks.

Check here for the source. I would love to see your feedbacks / suggestions / contributions to the library.

 

To the open-source community!!   

 

Happy Testing & Subscribe 🙂