Protractor is dead, long live Cypress! - Part 1

Table of Contents

• Landscape of E2E testing frameworks
  • WebDriver-Based
  • Chrome DevTools Protocol
  • Flakiness vs. Cross-Browser
• Why you should use Cypress
• Why you should not use Cypress
• Summary
• Further Reading

On 24th April, Angular announced the deprecation of their end-to-end (E2E) testing tool, Protractor. It remains unclear whether there will be a successor or if Angular will delegate this to its users. At the time of this writing, WebDriver.IO, TestCafé and Cypress have come up with schematics for the Angular CLI.

In this article, I will provide a short overview of the differences between the various E2E frameworks and explain why you should use Cypress. In the follow-up to this article, I will help you get started with Cypress.

If you prefer watching over reading, then this recording of my talk is for you:

Landscape of E2E testing frameworks

WebDriver-Based

We can roughly split the available tools into two groups. Those based on WebDriver and those that operate on the Chrome DevTools Protocol (CDP).

Selenium was released in 2004. It merged with a similar tool called WebDriver. Selenium is still a major player and Protractor is, more or less, a wrapper around it.

With Selenium/WebDriver, developers have a common API to automate a browser. Every supported browser provides a driver which WebDriver uses to make these frameworks cross-browser capable. This means that you can run the same test on Safari, Chrome, mobile browsers, etc.

Over time, WebDriver, the core technology in Selenium, became a W3C standard. That opened doors for other vendors to create their own frameworks based on WebDriver.

Unfortunately, WebDriver-based tests have a flaky reputation. If you run the tests multiple times under the same circumstances, they may fail or they may succeed. That is extremely bad for a testing framework. You spend a considerable amount of time building tests that should guarantee the stability of the application and then you can't even trust the tests’ results.

Chrome DevTools Protocol

Two quite popular alternatives are Puppeteer and Playwright. They don’t rely on WebDriver but talk directly to the browser via the Chrome DevTools Protocol (CDP). This gives them much better control which leads to more stable tests.

CDP binds to the Chromium-based browsers including Chrome and Edge. Although Firefox supports CDP as well, the kind of cross-browser testing WebDriver enables is not possible.

Some of Puppeteer’s original developers are behind Playwright. So one could see Playwright as a rewrite of Puppeteer. It uses CDP as well but also supports Safari because it ships with patched browsers.

WebdriverIO is a hybrid framework. It provides both access to CDP and WebDriver.

In which category does Cypress fall? Cypress relies on CDP but, in contrast to Puppeteer, Cypress only uses CDP to manage the tests. The tests themselves don't use CDP at all but run directly in the browser. There is no layer between the tests and the browser anymore. Your test code has direct access to the DOM and not an indirect one via CDP or WebDriver. Test and application code are one.

As it happens, that's the same approach that Selenium took before it merged with WebDriver. Looks like history repeats itself.

Flakiness vs. Cross-Browser

From a very high-level view, the more abstraction we have between our tests and the browser, the flakier tests tend to become. On the other hand, abstraction makes it easier to do cross-browser testing. The closer we get the browser the more stable the tests become. However, we must accept that we can only run the test in one particular browser technology, i.e. Chromium.
In the future, all browsers might support CDP or the upcoming WebDriver BiDi will bring stability into the WebDriver world. But things are still evolving.

Why you should use Cypress

First, Cypress does not use WebDriver but runs its tests directly in the browser. These make the tests stable and reliable. Exactly what we wanted in the first place.

The second reason is the development experience (DX). Cypress' authors invested a lot of effort in making it as easy to use as possible. This does not just mean that the API is easy to use. The tooling, the community work, and especially the documentation are just fantastic.

OK, so Cypress is better than Protractor. Why should you pick Cypress over any other E2E library?

As Angular developers, we always appreciated that we got an "opinionated framework" which pre-selected various tools for us. Although it might look like Angular leaves the spot for E2E empty, there is another big player: Nx has been providing the best tools for Angular applications for quite some years.

Nx is a wrapper around the CLI. It started out as a candidate for MonoRepositories but over the years became more and more of an Angular CLI++. In fact, I personally would always pick nx, if I start a new Angular application. There is no lock-in and no overhead. I just get better tooling and can stay with the Angular CLI.

And guess what? Nx replaced Protractor with Cypress. So nx takes care that Angular and Cypress work well together.

The last argument is the usage of Cypress among Angular developers. Before the deprecation was made public, Angular did a survey. They asked their users what E2E they are using. That survey showed 64% for Cypress. You call that a landslide victory in politics. And please remember, that was at a time when Protractor was the official tool! Protractor ended up with only 19%.

Why you should not use Cypress

Depending on your use case, Cypress might not fit your needs:

• No cross-browser testing: If you need to run your e2e tests in Safari, other browsers or even on mobiles devices, you need to use another tool. Cypress is definitely the wrong choice.
• Switching domains within one test: If your applications spans across multiple domains, you will also not be happy with Cypress.
• Switching tabs: Also not possible.

This is just a logical consequence if you think about how Cypress works. Tests run inside the browser and vanish if you change the URL or the tab.

Summary

Cypress gives you both stability and a great developer experience. The majority of Angular developers use it already.
If you are running E2E in Protractor, now is the time to migrate! If you haven't written E2E until now, take a look at Cypress. You will be surprised how easy it is.
Switching to Cypress means rewriting your E2E tests. You can do that in an incremental way. Cypress and Protractor don't exclude each other. They can be run in the same Angular project.
Be aware that Cypress might not fit your needs. If cross-browser support is an absolute must, you should check out alternatives like WebDriverIO and others mentioned above.

Outlook

In the second part, we will do our first steps with Cypress.

Further Reading

• WebDriver/RemoteProtocol
• W3C WebDriver protocol
• Future of Angular E2E & Plans for Protractor · Issue #5502 · angular/protractor
• Automation Protocols
• Playwright vs. Puppeteer
• Cypress vs. Selenium: Why Cypress is the better option
• What is Selenium RC : Difference from Webdriver
• Nightwatch.js | Node.js powered End-to-End testing framework
• Cypress vs. Other Test Runners
• Book: Learning Selenium Testing Tools – Third Edition, Raghavendra Prasad MG