Image comparison in automated testing
powered by ImageMagick
Latest trends suggest that image comparison in automated testing is becoming popular due to growing number of mobile devices and complexity of their UI. At almost every Selenium-related event, speakers mention such tools such as Applitools Eyes or Sikuli, even more so at Appium-related conferences and meetups. So, why is everyone excited about image comparison?
One reason to use image comparison is that ordinary tools we normally use have very limited capabilities in terms of UI validation. For example, using Selenium and Appium, we cannot verify icons, font sizes, and styles. Another problem is that writing a functional test that validates UI requires a lot of code. A lot of code means literally a ton of code as we have to describe position and size of each and every element, color and text content. Alternatively, we can just take a screenshot and compare it against last known good version.
There are a number of ways to compare images programmatically. Some of them have Java implementation, some are implemented in other languages, and using them with Java code can be complicated. But there is an open source command line utility ImageMagick, that operates both files and streams (pipes), can be used on all popular platforms, and is being actively used and supported. It powers many websites to resize and crop images and avatars, apply filters, and do many other transformations. At the very beginning, we were just taking a screenshot, loading a baseline image from resources, and comparing against each other.
Very quickly it became obvious that the system clock is ticking and notifications keep popping up. We needed some mechanism to ignore some things while still comparing other. ImageMagick provides such a mechanism, that ignores only pixels that are completely transparent. But how does one make only certain areas of the image transparent? ImageMagick allows you to apply one image as an opacity mask to another image. The only thing we need to do is just to set the color of the pixels to be ignored to black while pixels that should be compared must be white.
Here is a sequence of actions our library performs to compare two images and get a third image highlighting differences.
1. Copy screenshot (or original image) into a new file (it will be used later as a mask) and paint the whole mask image white color
convert screenshot.png -fill white -colorize 100% mask.png
2. Add rectangles representing ignored areas to the mask image (rectangles must be filled with black color)
convert mask.png -fill black -draw "rectangle 0,0 480,32" mask.png
3. Apply the mask image as an opacity mask to both screenshot and original image
convert original.png mask.png -compose copy-opacity -composite original.png
convert screenshot.png mask.png -compose copy-opacity -composite screenshot.png
4. Compare the two images and produce delta image
compare original.png screenshot.png delta.png
5. Remove opacity (this extra step is needed, otherwise ignored areas will not be visible)
convert original.png -alpha off original.png
convert screenshot.png -alpha off screenshot.png
6. Save all three images to disk
If we were using files to store images at each stage, it would easily consume a significant amount of space. Moreover, disk operations are slow and may slow down test execution. Luckily, ImageMagic can read images from stdin and write results to stdout. So, we only need to write images to disk at the very final stage of comparison, when results are written to HTML report.
Since we implemented automated visual testing for our application we have captured numerous visual issues, that otherwise would be probably missed. It is not able to completely replace functional testing, but it can be a good addition to it. Such tools, as Applitools Eyes, can be easily integrated with Selenium and Appium tests without investment into development and infrastructure. We have built our own library based on ImageMagic and written in Kotlin (but compatible with any JVM language).
We have published the library we have built on top of ImageMagick on GitHub: HotwireDotCom/image-assert
Feel free to use it, clone it, fork it and contribute back!
Any feedback or suggestions are very welcome.