Ocular – Automated Visual Validation for Selenium WebDriver Test Automation Frameworks

67 Comments / Articles, Extend WebDriver, Framework, Ocular, Selenium, Utility / By /

Overview:

Visual Validation is a process of validating the visual aspects of an application’s user interface – Verifying each element in the application is appearing exactly in the same size, color, shape etc.

Lets say, your application has charts to represent some data to the end user, how can we verify if the chart is appearing as expected?

Verifying the visual aspects using automated scripts is tedious. We might have to write tons of assertions! Mostly, automation engineers find this difficult to automate & take care of the application’s functionality testing and leave the UI related testing to the manual testers.

Even if the application layout changes, if the functionality works fine, automated test scripts never catch those issues!  Enhancing the automated tests to include the automated visual validation feature for an application is also very challenging!

 

Challenges in Visual Validation:

  • Commercial visual validation tools are very expensive – (We were given a quote of $100K per year by one of the famous vendors – for one of the applications)
  • Organizing / maintaining the baseline images for comparison for each pages / components of the application
  • Responsive Testing – Same page appears in different layouts in different devices
  • Reporting the differences
  • Applications tend to change over time – Visual validation tests are always failing!
  • Modifying existing framework is very difficult to include the visual validation feature.

 

TestAutomationGuru has come up with its own library, Ocular, for your selenium WebDriver test automation frameworks which could solve most of these challenges. It is FREE & Open Source!

Github:

Check here for the source.

Ocular:

Ocular is a simple java utility which helps us to add the visual validation feature into the existing WebDriver test automation frameworks.

(Ocular is developed on top of Arquillian RushEye which is a command-line image-comparison tool. Special thanks to Mr. Lukas who created the rusheye project & Arquillian’s team for maintaining it.

If you are an arquillian-graphene user, I am working on a Graphene Extension to include this ocular. I will update here when it is ready to use.)

Maven Dependency:

Place below artifact configuration into Maven dependencies just below your selenium-java dependency:

<dependency>
    <groupId>com.testautomationguru.ocular</groupId>
    <artifactId>ocular</artifactId>
    <version>1.0.0.Alpha</version>
</dependency>

Terms:

Ocular uses below terms.

  • Snapshot – the baseline image.
  • Sample – the actual image.
  • Similarity – how similar the sample should be while comparing against snapshot.

Global Configuration:

Ocular expects the user to configure the following properties before doing the visual validation.

Configuration Property Description Type Default Value
snapshotPath location of the baseline images Path null
resultPath location where the results with differences highlighted should be stored Path null
globalSimilarity % of pixels should match for the visual validation to pass int 100
saveSnapshot flag to save the snapshots automatically if they are not present. (useful for the very first time test run) boolean true

Code example:

Update Ocular global configuration as shown here once – for ex: in your @BeforeSuite method.

Ocular.config()
    .resultPath(Paths.get("c:/ocular/result"))
    .snapshotPath(Paths.get("c:/ocular/snpshot"))
    .globalSimilarity(95)
    .saveSnapshot(false);

Note: Ocular does NOT create these directories. Please ensure that they are present.

Snapshot in Page Objects:

Page Object has become the standard design pattern in test automation frameworks as it helps us to encapsulate/hide the complex UI details of the page and makes it easy for the client/tests to interact with the Page. Ocular makes use of the Page Object structure to keep the baseline images organized.

  • Using @Snap annotation, Page objects / Page abstractions are mapped to the baseline(snapshot) images for the visual validation.
  • The baseline images are expected to be available relative to the snapshotPath.

Example:

Lets assume, Ocular configuration is done as shown below.

Ocular.config()
    .resultPath(Paths.get("c:/ocular/result"))
    .snapshotPath(Paths.get("c:/ocular/snpshot"))
    .globalSimilarity(99)
    .saveSnapshot(true);

A Page Object is created for the below page as shown here.

RichFacesTheme-classic

 

@Snap("RichFace.png")
public class RichFace {

    private final WebDriver driver;

    public RichFace(WebDriver driver){
        this.driver = driver;
    }

}

Now Ocular looks for the snapshot image at this location – c:/ocular/snpshot/RichFace.png – (snapshotPath + Path given in @Snap)- in order to do the visual validation.

 Note: If the image is not present in the location, Ocular creates image and place it under snapshot path. So that Ocular can use this image from the next run onward for comparison.

Comparing Snapshot against Sample:

Sample is the actual image. Sample is created by taking the screenshot of the current page using the WebDriver.

@Snap("RichFace.png")
public class RichFace {

    private final WebDriver driver;

    public RichFace(WebDriver driver) {
        this.driver = driver;
    }

    public OcularResult compare() {
        return Ocular.snapshot().from(this)     (1)
                     .sample().using(driver)    (2)
                     .compare();                (3)
    }
}
  1. snpshot.from(this) – lets Ocular read the @Snap value
  2. sample.using(driver) – lets Ocular to take the current page screenshot
  3. compare() – compares the snapshot against sample and returns the result

Different ways to access Snapshot:

Ocular.snapshot()
    .from(this)     // or
    .from(RichFace.class) // or
    .from(Paths.get("/path/of/the/image"))

Choosing Right Snapshots at Run Time:

Class is a template. Objects are instances. How can we use a hard coded value for @Snap of a Page Class for different instances? What if we need to use different baseline images for different instances? May be different baselines different browsers or different devices. How can we do the mapping here?

Valid Question!!

Lets consider the below RichFace page for example.

RichFacesTheme-classic

If you notice, this page has different themes – like ruby, wine, classic, blue sky etc. Same content with different themes. So, We can not create different page classes for each theme. Ocular provides a workaround for this as shown here!

@Snap("RichFace-#{THEME}.png")
public class RichFace {

    private final WebDriver driver;

    public RichFace(WebDriver driver){
        this.driver = driver;
    }

    public OcularResult compare() {
        return Ocular
            .snapshot()
                .from(this)
                .replaceAttribute("THEME","ruby") // lets the ocular look for RichFace-ruby.png
            .sample()
                .using(driver)
            .compare();
    }
}

You could use some variables in the @Snap , get them replaced with correct values.

Excluding Elements:

Sometimes your application might have an element which could contain some non-deterministic values. For example, some random number like order conformation number, date/time or 3rd party ads etc. If we do visual validation, We know for sure that snapshot and sample will not match. So, We need to exclude those elements. It can be achieved as shown here.

Ocular.snapshot()
        .from(this)
    .sample()
        .using(driver)
        .exclude(element)
    .compare();

If we need to exclude a list of elements,

List<WebElement> elements = getElementsToBeExcluded();

Ocular.snapshot()
        .from(this)
    .sample()
        .using(driver)
        .exclude(elements)
    .compare();

or

Ocular.snapshot()
        .from(this)
    .sample()
        .using(driver)
        .exclude(element1)
        .exclude(element2)
        .exclude(element3)
    .compare();

Comparing One Specific Element:

Ocular can also compare a specific element instead of a whole web page.

Ocular.snapshot()
        .from(this)
    .sample()
        .using(driver)
        .element(element)
    .compare();

Similarity:

Sometimes we might not be interested in 100% match. Applications might change a little over time. So we might not want very sensitive compare. In this case, We could use similarity to define the % of pixels match. For the below example, If 85% of the pixels match, then Ocular will pass the visual validation. This will override the Ocular.config().globalSimilarity() config settings for this page object / fragment.

@Snap(value="RichFace-#{THEME}.png",similarity=85)
public class RichFace {

}

You could also override global similarity config value as shown here.

@Snap("RichFace-#{THEME}.png")
public class RichFace {

    private final WebDriver driver;

    public RichFace(WebDriver driver){
        this.driver = driver;
    }

    public OcularResult compare() {
        return Ocular.snapshot()
            .from(this)
            .replaceAttribute("THEME","ruby") // lets the ocular look for RichFace-ruby.png
        .sample()
            .using(driver)
            .similarity(85)
        .compare();
    }
}

Ocular Result:

OcularResult extends ComparisonResult class of arquillian-rusheye which has a set of methods & provides result of the visual validation. The image with differences highlighted is getting saved at Ocular.config().resultPath() every time!

OcularResult result = Ocular.snapshot().from(this)
             .sample().using(driver)
             .compare();

result.isEqualImages() // true or false
      .getSimilairty() // % of pixels match
      .getDiffImage()  // BufferedImage differences highlighted
      .getEqualPixels() // number of pixels matched
      .getTotalPixels() // total number of pixels

Sample Diff Image:

Ocular creates below diff image in case of snapshot & sample comparison mismatch highlighting the differences between them.

RichFaces-Poll

Responsive Web Design Testing:

Ocular can also be used to test the responsive web design of your application.

responsive

To the responsive web design, Lets organize our snapshots in our test framework as shown here.

src
	test
		resources
			ocular
				iphone
					PageA.png
					PageB.png
					PageC.png
				ipad
					PageA.png
					PageB.png
					PageC.png
				galaxy-tab
					PageA.png
					PageB.png
					PageC.png					
				1280x1024
					PageA.png
					PageB.png
					PageC.png

Lets update the Ocular global config with the path where Ocular should expect the snapshots for the visual validation.

String device = "ipad";
Path path = Paths.get(".", "src/test/resources/ocular", device);
Ocular.config().snapshotPath(path);


// now Ocular will use the PageB.png under ipad folder.                 
@Snap(value="PageB.png")
public class PageB {
 
}

Now as shown above we could easily incorporate the responsive web design validation in our test framework.

Summary:

Ocular, with its static and fluent API, makes its easy for us to use the library & include the automated UI / Responsive web design testing feature in our test frameworks.

Check here for the source. I would love to see your feedbacks / suggestions / contributions to the library.

 

To the open-source community!!   18217-cheers1

 

Happy Testing & Subscribe 🙂

 

 

Share This:

67 thoughts on “Ocular – Automated Visual Validation for Selenium WebDriver Test Automation Frameworks”

  1. Hi,

    Thanks for this library.

    In the difference image. Is it possible to highlight part of base image & the actual image in different colors. It will really have good readability. Currently I see that only red color is being used.

    Reply

    1. Kumar,

      It can be done. In fact I was thinking of showing the area which differs side-by-side. But it requires some work. I will work on that when I find time.

      Reply

  2. Hello,
    thank you very much for sharing this, i have made a few tests and i think this is a very good tool.
    i have a windows machine and everything works good locally, but when i try to run it on my linux server i get an exception on the comparison save image, any changes i need to apply for this to run on linux ? . i will add the error:

    com.testautomationguru.ocular.exception.OcularException: Unable to write the difference
    Caused by: javax.imageio.IIOException: Can’t create an ImageOutputStream!

    Reply

  3. Thank you , my problem was that the Ocular result directory was empty when i committed to git, and i guess it does not include an empty folder. so i just added a file there and it was added and all is working,
    thanks again for a good project.
    Ariel

    Reply

    1. Ankit, Ocular can do the visual validation for a specific element instead of whole page – Please check the article section – Comparing One Specific Element

      Reply

  5. Hi Vinoth,

    Thanks for the useful tool. Have few queries. How to use Ocular in other frameworks other than Page Object Model?
    ie., During my test flow, can I pass the current driver snapshot to Ocular.compare() method for the validation? Basically everytime I call the Ocular.compare() method, is it possible to pass the expected snapshot and actual snapshot?

    Thanks.

    Reply

    1. Umai, Ocular does not have to be part of the page object. But I would prefer to do that as it seems to be easy for me to maintain. You can try this way to directly send the image file path.

      Ocular.snapshot().from(Paths.get("....path of the image file"))
      .sample().using(Paths.get("....path of another image file"))
      .compare();

      Reply

  6. Seems like taking the snapshot of particular element is not working properly….! I’m getting cropped images for my webElement….can you redirect me to the source code of taking only particular element screenshot?

    Reply

    1. We have been using this for few projects in our org. Both element level comparison and exclusion work fine for us. Can you upload a sample script for me to take look?

      Reply

  7. During further investigation, I found out that in case I’m running my tests on Mac-Chrome then it’s not taking correct snapshot. Works fine when run against Windows-Chrome. Wondering what could be the issue?

    Reply

    1. Ocular depends on Selenium WebDriver to provide the screenshot. The issue you mention is already existing in Selenium-WebDriver for mac with retina display – you could find that in open issues.

      Reply

  8. Can sombody please provide a complete sample code on how to do an image validation. I am new to visual automation & still difficult to understand how can we apply this in a test suite

    Reply

      1. Thank you . I followed mentioned example & I am getting below error. Can you please assist me to resolve
        java.lang.NoClassDefFoundError: org/arquillian/rusheye/comparison/ImageComparator

        Reply

  10. Hi,
    I am trying to exclude dynamic element(Date and time) in my portal by using below code
    OcularResult result = Ocular.snapshot().from(Paths.get(“C:\ocular\Baseline\NewTemplatePage.png”)).sample().using(driver).excluding(Commonobj.GetWebElement(“DateTime_XPATH”)).compare();

    But, still result snapshot is highlighting the difference of date and Time .
    Is there anything I am missing here?

    Thanks,
    Sreekanth

    Reply

    1. What does it return – Commonobj.GetWebElement(“DateTime_XPATH”)? The behavior you mention will happen only if you pass an empty list to the excluding method.

      Reply

  11. Thank you so much for the tool, very useful and easy to use.
    Please help me with how to access sample file for further use. for example, when images are not equal, i would like to print expected and actual screenshots in my results.

    Reply

    1. Please check the ‘Global Configuration’ section of this article. resultPath is the place where the difference image file is stored when it does not match with baseline.

      Reply

  12. Thank you for tool. I took this tool on try and I ran on the problem which I can’t solve.
    java.awt.image.RasterFormatException: (y + height) is outside of Raster
    Maybe you can tell me how I can resolve this?

    Reply

    1. Christian, The issue comes from Selenium. Not Ocular. If you are using mac (may be because of its resolution) – when you get the element’s image, it seems to get the subimage outside the image boundary. If possible give a try after changing the resolution to 1920×1080.

      Reply

      1. I changed the resolution, but this not helped me :/.
        This is funny situation. When I used className X – not worked, after I changed on another className still on this same page and snap was created. You had that situation in your job?

        Reply

      1. Thank you team, Its one of the best available open source tool online to do such comparisions.
        Got it working in katalon 🙂

        However, I think Testing a page’s responsiveness wouldn’t be achieved by this as it compare images which lead me to a situation where the elements where there on the page but was misaligned and comparison failed
        And excluding elements will be tedious and unfeasible for such case.

        Reply

          1. Thank you for your quick response 🙂
            My concern is very straight forward,
            I am looking to achieve this across different browsers i.e. Opening google.com in chrome and comparing the same in FireFox and IE as we know the element remains the same over there but the alignment of it changes which results in assert failure.

            I am not sure if this can be possible.

          2. It does not seem to be responsiveness testing. You want to do cross browser testing. If they are looking differently due to alignment issues , is it not supposed to fail?
            If you are ok for some misalignment – that is – not to do very sensitive compare, we have a workaround by using similarity.

            Similarity – by setting it to 95% – you mean to say you are ok some mismatch as long as it is within 5% of the total pixels.

          3. Forgot to add, ensure that you set same resolution to the browser windows.

  14. Hi
    I see that color comparison is also supported using Ocular. But the sample diff image is in black and white color making it difficult to know the difference between the baseline and actual images. Can someone please help me out how to deal with this.

    Reply

    1. The difference would be in red color highlighted. If you see pure black and white image, then there is no difference! You should look at the difference if the isEqualImages method returns false.

      Reply

  15. Hello, Thanks for the quick reply.
    I see that the difference is being highlighted in red color. But we would not know what is the difference if sample diff image is in white and black.
    For eg: Let say an element is in red color in baseline image. And while running the test, the element is in green color. The sample diff image highlights the element in red color, but since the image is in white and black, I would not know why the test is failed and what color the element was in when the script ran.

    Reply

    1. You basically mean to save the actual image + diff image, rt? if yes, it is not possible as of now.

      Reply

  16. Thank you for tool. I used this in my project its working fine in some places but in some places i got the below exception.
    java.awt.image.RasterFormatException: (y + height) is outside of Raster

    Please let me know how can i solve this problem?

    Reply

        1. I have seen this issue before. It is not specific to Ocular. If we try to use some non-standard screen resolution / try to get screenshot of a partially visible web element etc..we might face this issue from selenium lib.

          Reply

  18. Hi, Its a great tool. Just started to play with it. Wondering whether it supports taking the full screenshot or just viewable size ?

    Reply

  20. Thanks for the tool. I just play with if have concerns:
    1. How can I compare element which is at the bottom of a long screen (need to scroll down)?
    2. It seems that the tool create the baseline image just on top of a page, can we add more base line images of a page by scrolling?

    Reply

    1. Ocular currently compares elements in the view port. So you need to focus/scroll yourself using javascript – element.scrollIntoView(true);
      You could have N number of elements and have baseline for all the elements. Check the Choosing Right Snapshots at Run Time: section on using dynamic attributes.

      Reply

    1. If you think that image would keep changing, i would say ignore that element and compare other elements. Aim of visual validation is to check if the application shows what you are expecting it to show.

      Reply

  24. Hey , Its fantastic. Waiting for different color result . And it would be better if you can provide generic solution to scroll the page and take screenshot,

    Reply

  25. Hello! Do i understand correctly, that i can’t simply check full scrolled till the end page (not just top part of it) ?

    Reply

  26. Ocular.snapshot().from(Paths.get(“C:\ocular\results\1.png”))
    .sample().using(Paths.get(“C:\ocular\snapshot\2.png”))
    .compare();
    when i am comparing two images i am getting “java.lang.NullPointerException”
    please help

    Reply

  27. I was trying to use this and I figured out few issues which were due to the dependency used by it.
    Dependency (group: ‘org.arquillian.rusheye’, module: ‘rusheye-impl’) uses the older version of common.io (group: ‘commons-io’, module: ‘commons-io’, version:’1.4′) which results in conflict.

    New version of commons-io is available and the version is 2.6

    Reply

  28. Hi guys .. Have you tried capturing the image using x and y co-ordinates or weight and height ??
    I have a canvas graph to automate where my x axis changes based on time. So i want to exclude the x axis part. Since this is canvas element , i can not exclude using webelement.

    Any help would be appreciated guys .. Thanks in advance

    Reply

  29. Kindly confirm guys. whether ocular it is working or not. i have tried the pom using java. but ocular library not working.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.