DEV Community

Cover image for Effective Visual Regression Testing for Developers: Vitest vs Playwright
Maya Shavin πŸŒ·β˜•οΈπŸ‘
Maya Shavin πŸŒ·β˜•οΈπŸ‘

Posted on • Originally published at mayashavin.com

Effective Visual Regression Testing for Developers: Vitest vs Playwright

Visual regression testing plays a crucial role in ensuring the UI and UX's consistency across different browsers, devices, and even screen sizes, especially for large applications. In this post, we will covers snapshot and pixel-to-pixel comparison with Vitest and Playwright, explore limitations, and provide insights into choosing the right visual testing approach for your project.

Table of Contents

Prequisites

For the demo purpose, we will use a Vue 3 project with Vitest and Playwright installed, with a SearchBox component as follows:

Search input box component

The implementation of the SearchBox component is available in this post.

For React developers, you can replace the Vue component with a React one, and Vue Test Utils with React Testing Library to render component. The rest of the steps below remains the same.

What is visual regression testing?

Visual regression testing (or visual testing) is a technique to verify the visual appearance and usability of an application's UI between changes, helping to identify any unintended bugs that might occur to the existing functionalities during the development process. It focuses solely on validating the visual aspects of a component that a user sees or interacts with, including layout, styling, and other visual elements.

There are several types of visual testing, including DOM-based comparison (snapshot testing), pixel-based comparison and manual comparison using screenshots. Depending on the requirements, we can implement visual testing at different levels with different tools, such as unit testing with Vitest or E2E testing with Playwright, or combining the two.

We'll start with visual unit testing using Vitest.

Using snapshot for visual regression comparison in Vitest

Vitest provides a toMatchSnapshot() method to take a snapshot of the component and compare it with an existing one, as seen in the following test for SearchBox Vue component:

import { render } from '@vitest/vue'
import SearchBox from './SearchBox.vue'

describe("SearchBox component", () => {
  it('should match snapshot', () => {
    const wrapper = mount(SearchBox, {
      /** props and other global plugins */
    });

    expect(wrapper).toMatchSnapshot();
  })
})
Enter fullscreen mode Exit fullscreen mode

Upon the first run, Vitest will take a snapshot of the SearchBox component and store it in a .snap file within the __snapshots__ folder:

Screenshot of the snapshot file's location

Below shows an example of the snapshot file, containing the component's DOM structures and HTML layout:

Screenshot of how a snapshot file for SearchBox looks like

In the next run, Vitest will generate a new snapshot and compare it with the stored one. If they are different, for example, when we modify the input's placeholder to Search for a beer, the test will fail and Vitest will highlight the difference will in the terminal as follows:

Screenshot of how a snapshot file for SearchBox looks like

Base on this result, we can decide whether to re-visit the changes, or update the snapshot file with the -u flag in the Vitest run command to accept them.

With that, we have enabled snapshot visual testing for our SearchBox component. However, there is a downside of using this method, which we will discuss in the next section.

Limitations of snapshot visual testing

Regular snapshot visual testing, unfortunately, does not fully provide a great developer experience, and high reliability level in visual consistency. It only compares the DOM structure and basic HTML layout of the component generated, without considering other styling aspects like colors, fonts, etc. For instance, if we change the input field #searchbox's color from black to red:

<style>
#searchbox {
  color: red;
}
</style>
Enter fullscreen mode Exit fullscreen mode

The snapshot test will still pass, since it is unable to validate the style changes of the component. This limitation makes it less reliable for visual testing, especially for components with dynamic content or styles.

Another limitation is the readability of the snapshot files, which can lead to a maintenance nightmare, as developers may find it hard to identify the changes in these files for large components. After all, visual means what you see, and this snapshot testing does not provide a proper "see" representation.

A better alternative is using screenshot-based visual testing with Playwright, which we will discuss next.

Pixel-to-pixel screenshot testing in Playwright

To perform an E2E visual regression validation, Playwright provides a toHaveScreenshot() method that takes and compares screenshots of a specific component (in component testing), an element, or the whole page, as seen below for our SearchBox component:

/** e2e/searchbox.spec.js */
import { test, expect } from '@playwright/experimental-ct-vue';
import SearchBox from "../src/components/SearchBox.vue";

test("should match screenshot", async ({ mount, page }) => {
  const component = await mount(
    <SearchBox searchTerm="hello" />
  );

  await expect(page).toHaveScreenshot();
  await expect(component).toHaveScreenshot();

  /** other assertions */
})
Enter fullscreen mode Exit fullscreen mode

On the first run, Playwright captures and stores the screenshots as an .png files to the __snapshots__\searchbox.spec.js-snapshots folder, with each file named based on the test case's name (should match screenshot), the testing browser (chrome) and the platform (darwin), as follows:

Screenshot of the screenshot files' location

The generated screenshot file contains the view of the full page (1):

browser's page contains an input field with hello as initial value

And of the component (2), as follows:

input field with hello as initial value

The runner also throws an error. From the second run onwards, Playwright will compare the new screenshots with the current stored ones, and highlight the pixel difference (with ratio) in the terminal, such as when we change the input's font color to red:

failed screenshot comparison with 65 pixels difference

An image of the expected result, the actual result, and the difference between them is also available in the test result dashboard in the browser:

actual vs expected side by side

Additionally, we can adjust different the screenshot matcher's configurations like the maximum pixel difference maxPixelDifference, and threshold level by providing them as options to the toHaveScreenshot() method as follows:

await expect(page).toHaveScreenshot({
  maxPixelDifference: 100,
  threshold: 0.1,
});
Enter fullscreen mode Exit fullscreen mode

With the above settings, Playwright will allow a maximum of 100 pixels difference and a threshold of 0.1 for the screenshot comparison. This way, we can fine-tune our visual testing to meet the project's requirements.

toHaveScreenshot() method is a great tool, as it provides a pixel-to-pixel visual comparison for the component. However, it only works with native Playwright tests, and isn't supported in Vitest's browser mode. For that, we need to use screenshot() method in the next section.

Screenshot visual testing with Vitest's browser mode and Playwright

We first need to install the relevant packages and enable the browser mode in Vitest, following this tutorial.

Once enabled, Vitest will then use Playwright to run all the tests in browser mode. Similar to the native Playwright, within a test, we can use screenshot() method to take a screenshot of a specific element, such as input, and save it to the path provided:

/**... */
  test('renders search input', async () => {
    /**... */
    const input = component.getByTestId('search-input')

    await input.screenshot({
      path: 'screenshots/searchbox_default_full.png',
    })
  });
Enter fullscreen mode Exit fullscreen mode

The above code results in a screenshot of the input field, as shown below:

input field with hello as initial value

For the page's screenshot, we use the page object, from @vitest/browser/context, which is similar to the native Playwright's page object:

import { page } from '@vitest/browser/context';
/**... */
  test('renders search input', async () => {
    /**... */
    await page.screenshot({
      path: 'screenshots/searchbox_page.png',
    });
  });
Enter fullscreen mode Exit fullscreen mode

Similar to toHaveScreenshot(), we can also adjust the screenshot's configuration by providing additional options to the screenshot() method, such as type, clip, omitBackground, etc., to define the file format, or capture the component's specific area.

Upon running the test, Vitest will take the screenshots and store them accordingly, ready for us to manually verify the UI, or integrate with an analyzer such as Applitools or Percy for visual comparison in an automated workflow.

But that's also the limitation of using Playwright's screenshot(), as it requires manual verification or integration with third-party tools, which can generate additional costs. At the time of writing, Vitest browser mode does not support Playwright's toHaveScreenshot() method or toMatchImageSnapshot() matcher from Jest-Image-Snapshot, which would have been a better option.


Summary

We have explored briefly different methods of visual testing with Vitest and Playwright, from manual to automated visual comparison with snapshot and screenshot, and the pros and cons of each approach. It is essential to consider your project's requirements and budget for choosing the suitable testing approach and keep the process efficient.

What's next? How about trying it out yourself and see which method works best for your project? Let me know your thoughts in the comments below!

πŸ‘‰ Learn about Vue 3 and TypeScript with my new book Learning Vue!

πŸ‘‰ Follow me on X | LinkedIn.

Like this post or find it helpful? Share it πŸ‘‡πŸΌ or Buy me a coffee

Top comments (0)