In one of our client projects, we had to decide on an end-to-end (e2e) testing framework. While we were familiar with Cypress, we wanted to explore a newer option, Playwright. We compared how both frameworks addressed the issues that were most important to us, allowing us to establish criteria for making an informed decision. I'd like to summarize some of the key points we discussed and the conclusions we reached.

1. Installation

Both frameworks provide npx scripts that manage package installation and create example tests to help users get a feel for the framework. They also offer Docker images that can be used for CI integration. While all necessary dependencies for Playwright are stored in node_modules (with browsers included in the binary), Cypress relies on what is provided in the system (or Docker image). It's worth mentioning that the Cypress binary is stored globally in the system, which can complicate installation if you're working on a corporate PC with restricted rights to install software. This has another implication: if you wish to cache dependencies between CI jobs, you'll need to set the caching directory and store it next to node_modules.

2. Test Explorer

A comprehensive test explorer is essential for navigating and executing tests seamlessly. Both frameworks offer this functionality, but the developer experience (DX) varies. Cypress presents the real application in its explorer, allowing developers to click through the app and explore the tests once they finish execution. Playwright offers similar functionality, but instead of presenting the real application, it displays app screenshots on a timeline. This might feel like a limitation, but fortunately, there is a VSCode extension that helps overcome it.

3. Syntax and Execution Environment

Both testing frameworks support TypeScript. Cypress handles asynchronous events with a wrapper that looks similar to Promises, while Playwright uses native Promises and supports async/await syntax. It's important to note that Cypress code is executed in the browser, whereas Playwright runs in a 'Node' environment and communicates with the browser through the DevTools protocol. As a result, environment variables can be accessed directly at runtime, which might be useful for passing test-user credentials or other test-related data.

Below you'll find code snippets of what tests look like for both frameworks. Bear in mind that all selectors have been picked up automatically by the locator picker tool (available in both frameworks).

Example test written in Cypress. Cypress manages asynchronous tasks internally, which is why one of the console logs is logged out of order, right at the start of the test. This might be surprising at first. To execute it after some action, you need to wrap it around the then method.

Example Playwright test. The framework uses native async/await syntax, so each statement is executed in order, as one would expect.

4. Parallel Test Execution

Both frameworks support parallel test execution, but it's worth noting that this option is available in Playwright out of the box and for free. With Cypress, however, you need to subscribe to Cypress Cloud, which is only free for small teams.

5. API Testing

In our project, we wanted to cover some functionality with API tests. Our initial idea was to use 'Jest' for this purpose, but it would require us to implement a mechanism to manage auth tokens. With either framework (Playwright or Cypress), this process is simplified. The token can be acquired at the beginning of a test session using application logic (since we have access to the real application) and stored. Later, it can be reused for each API test.

Example API test using Cypress. Cypress uses the Chai assertion library. You might want to extend it with additional modules, allowing for deep checking arrays or checking the subset of properties in the result object.

Example API test using Playwright. For those familiar with Jest, Playwright's syntax looks familiar. It has built-in generic assertions, a class that lets us check if a received object contains a specific value, if a value is close to another, or if a value is of a specific type, and so on.

6. Different Browser Tabs, Changing Origin During Test

Simulating real-world scenarios often involves testing across multiple browser tabs or changing the origin during test execution. While Cypress has a workaround for the latter, it's impossible to execute tests involving changing browser tabs. Playwright, on the other hand, handles such tasks effortlessly due to its method of communicating with the browser. You can open new tabs and switch between them within a test case easily.

7. Mobile Viewports

Both Cypress and Playwright support testing on mobile viewports, offering predefined resolutions for various devices. Playwright's list seems to be longer than Cypress'. In both cases, the resolution can be set either globally (or per project in the case of Playwright) or within a test case. However, it's important to note that this is not about emulating a device, but a device resolution. For thorough testing in real-life scenarios, external tools like Browserstack or Lambdatest are necessary.

8. Visual Comparisons

If you want to ensure that your CSS isn't broken after a change, visual comparison tests are a good idea. Both frameworks support this option, but with different levels of adoption simplicity. In Playwright, it works out of the box; you simply call await expect(page).toHaveScreenshot();. This function generates a new screenshot if one isn't already defined; otherwise, it compares the current snapshot to what's stored in the project, with no additional setup required. Cypress, on the other hand, does not support this natively, so you need to rely on external plugins. I recommend checking the official Cypress docs to see what tools are recommended.

Summary

Taking everything into consideration, we decided to go with Playwright. Its ability to control the browser via the DevTools protocol is a significant advantage, as it allows for testing on different browser tabs without issues. Additionally, its syntax is more explicit, using native Promises that can be consumed with standard await, compared to the promise-like syntax of Cypress. We also appreciated how easy it is to perform visual testing with Playwright.

Miscellaneous

In our case, running tests on CI turned out to be quite slow. After some investigation, we discovered that the issue was a misconfiguration of our Playwright setup specifically, the trace viewer configuration. We had been using the retain-on-failure setting, which records traces for all the specs. This resulted in a large number of files needing to be stored, which was very time-consuming for our GitLab runner. Switching to the 'on-first-retry' option resolved the issue.