Visual Regression Testing - Demo using TestPlane

What is Visual Regression Testing?

Visual regression testing is like your superpower to catch those sneaky, unintended visual changes in your UI before they reach production. It’s the difference between saying, “I hope this doesn’t break anything” and knowing it won’t.

How It Works:

Visual regression tools are like high-tech spies for your UI. They take snapshots of your application before and after changes, comparing them to uncover any sneaky differences. Think of it as a high-stakes game of "Spot the Difference" with your app’s layout and design. Here’s the process in a nutshell:

Snapshot Time: Capture your UI’s look before changes (baseline snapshot).
Change Detection: Snap your UI again after changes (current release snapshot).
Comparison: The tool compares these snapshots, identifying any discrepancies.
Feedback Loop: If differences are intended, update the baseline image. If not, you’ll be alerted to potential issues.

Remember those “Spot the Difference” games from newspapers? Visual regression testing is like that, but supercharged. Imagine finding all those differences in seconds, effortlessly!

Ready to put your observation skills to the test? Let’s play a game!

Take a look at these two images:

They might look identical, but can you spot all the differences? Try your hand at it, and see how you compare to a visual regression testing tool. For a fun challenge, play the same game here.

See how it went?

I have marked all the issues which were reported by the tool!! Check out how the tool's findings matched up:

Result from tool :) All 14 issues matched!! Seems efficient.

Comments how many issues you find in the game..

The visual regression testing tool caught all 14 issues, proving its efficiency and reliability. Testing isn't just about catching bugs; it's a skill in observation and reporting. And now you’ve seen it in action!

As some wise man said,

The only way to go fast is to go well.

So this testing can help us in moving shift left and reduce our regression time.

Want to know Tool?

Now that you understand the concept of visual testing, let’s dive into how I use this tool.

A few years back, I explored a tool called Gemini by Yandex for visual regression testing. Although Gemini is deprecated now, and they created another open source tool TestPlane which we are discussing further and can be explored with the link given.

Let's see how we can install this tool on our local system

npm init testplane@latest demoProject

After installation you will get new folder called demoProject and in that folder you will see node_modules where all node packages installed will come and config file with name .testplane.conf.ts. I have added some additional configs in it (Example: Running suite on mobile and desktop chrome browsers).

export default {
    gridUrl: "http://localhost:4444/wd/hub",
    baseUrl: "http://localhost",
    pageLoadTimeout: 0,
    httpTimeout: 60000,
    testTimeout: 90000,
    resetCursor: false,
    sets: {
        desktop: {
            files: [
                "testplane-tests/**/*.testplane.(t|j)s"
            ],
            browsers: [
                "chrome-phone",
                "chrome"
            ]
        }
    },
    browsers: {
        "chrome-phone": {
            automationProtocol: "devtools",
            headless: true,
            desiredCapabilities: {
                browserName: "chrome",
                "goog:chromeOptions": {
                    args: ["--hide-scrollbars"],
                    mobileEmulation: {
                        deviceMetrics: {
                            width: 360,
                            height: 640,
                            pixelRatio: 3.0
                        },
                        userAgent: "Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 5 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Mobile Safari/535.19"
                    }
                }
            },
            sessionEnvFlags: {
                isMobile: true
            }
        },
        "chrome": {
            automationProtocol: "devtools",
            headless: true,
            desiredCapabilities: {
                browserName: "chrome",
                "goog:chromeOptions": {
                    args: ["--hide-scrollbars", "--start-maximized","--window-size=1920,1080"]
                }
            },
            windowSize: {
                width: 1920,
                height: 1080
            },
            sessionEnvFlags: {
                isMobile: false
            }
        }
    },
    plugins: {
        "html-reporter/testplane": {
            // https://github.com/gemini-testing/html-reporter
            enabled: true,
            path: "testplane-report",
            defaultView: "all",
            diffMode: "3-up-scaled"
        }
    }
};

We are defining desired capabilities in json format where we are defining browser properties as we defined in ChromeDriver initialisation in Selenium. You can read more about it here.

And here in example.testplane.ts we are asserting github testplane repository home page.

describe("Github Public Repository", () => {
    it("Home Page", async ({ browser }) => {
        await browser.url("https://github.com/gemini-testing/testplane");
        await browser.$("body").assertView("Full View Port",
            {
                allowViewportOverflow: true,
                captureElementFromTop: true,
                compositeImage: true,
                screenshotDelay: 1000
            }
        )
    });
});

If you see it is finding difference on last released items [18 hours] since our baseline image has [17 hours]. So it is giving failure on dynamic element which is blocker for using above code.

Since we have error on dynamic element it is observing correct but seems false positive as a tester we have information about what is dynamic or what is static.

You can define dynamic elements in ignoreElements

describe("Github Public Repository", () => {
    it("Home Page", async ({ browser }) => {
        await browser.url("https://github.com/gemini-testing/testplane");
        await browser.$("body").assertView("Full View Port",
            {
                allowViewportOverflow: true,
                captureElementFromTop: true,
                compositeImage: true,
                screenshotDelay: 1000,
                ignoreElements: [
                    "div[data-testid='latest-commit-details']",
                    "a[class='Link--primary d-flex no-underline']",
                    "div[class='react-directory-commit-age']"
                ]
            }
        )
    });
});

We can add fields like ignoreDiffPixelCount, tolerance, antialiasingTolerance, screenshotDelay if you have some specific requirements. If you see dynamic element is coming as black coloured which we define in ignoreElements. Commit and Release elements are coming black. With this we can ignore dynamic elements.

Now if we have some intended dev changes in the build and you want to change baseline image. You can go to accept mode and convert into baseline image, and then rerun the tests.

We can improve above code by defining types and constants in typescript for make it more reusable.

For executing test case use below command:

npx testplane gui

And that will open url mentioned in baseUrl defined in .testplane.conf.ts.

If you want to check baseline images testplane/screens/.
Above executable code can be found in repository

Moreover we can use this to visually test different elements as in above code we are testing whole body. In other case we can check any specific section then i.e also possible similarly.

If you have use cases related to Before test, Before suite or clean up at the end. You can check this resource.

Future Integration

Integrate TestPlane with your functional testing framework to check visual changes across:

• Different Browser Versions
• Different Browser Types
• Different Devices

Visual regression testing is not just about identifying bugs; it's about ensuring a seamless user experience.

Conclusion

Visual regression testing is a game-changer for maintaining the visual integrity of your UI. By integrating this testing technique into your workflow, you can catch unintended visual changes early, ensuring a seamless user experience and boosting confidence in your releases.

Key Takeaways:

• Spot the Differences: Visual regression testing tools act as vigilant eyes, identifying discrepancies between snapshots of your UI and alerting you to potential issues.
• Supercharge Your Testing: Just like those “Spot the Difference” games we loved as kids, visual regression testing allows you to find discrepancies quickly and efficiently.
• Streamline Your Workflow: Tools like TestPlane provide robust solutions for visual regression testing, integrating smoothly with your existing setup and offering powerful configuration options.
• Enhance Your Coverage: Beyond simple UI checks, visual regression testing can be integrated into various scenarios, including different browsers, devices, and screen sizes, ensuring comprehensive coverage.

I have used this tool in one of my project and I found it very efficient and it reduce lots of time and small issues which can be missed.

Got any questions or experiences with visual regression testing tools? Share them in the comments below!