What Cypress E2E testing has taught us about our code

A little over a year ago Forem introduced End to End testing to the app using Cypress. Since that time we've definitely evolved our understanding of Cypress, but we've also learned a lot about our app too. So much so, that I wanted to share some of the things Cypress testing has taught us 😄

Some background

We use Cypress alongside cypress-rails and Cypress Testing Library.

Testing Library gives us a range of enhanced queries that help us focus on testing in a way that closely mirrors how users interact with the app. Instead of targeting page elements with e.g. CSS selectors or IDs, we lean towards finding elements by their role and accessible name.

When I talk about testing with Cypress at Forem, I mean specifically testing with Cypress and Cypress Testing Library 😄

A greater awareness of accessibility

I've made no secret of personally being a huge fan of Testing Library for the impact it can have on accessibility testing and awareness. It therefore isn't a surprise to me that we have learned a lot about the accessibility of our app since introducing E2E tests with Cypress.

Uniqueness of elements

One of the challenges we've had with testing some older views in Cypress is discovering that the same element/name combination is present multiple times.

A great example of this is the "Follow" buttons throughout the app. We tried to target a Follow button in test with cy.findByRole('button', { name: 'Follow' } and found it incredibly difficult to target the one we wanted.

This immediately flagged itself up as an accessibility issue - users of screen readers would hear every follow button announced as "Follow"... but, follow who? 😅 Just as we were struggling to target the right button in test, a screen reader user would struggle to target the button of the user they wanted to follow.

Big shout out to community member @keshavbiswa for making improvements to the follow buttons accessible names after we discovered this!

The power of landmarks

We take advantage of Cypress' scoping abilities by writing code like:

cy.findByRole("main").within(() => {
  // Everything in here is scoped to the "main" element only
  cy.findByRole('heading', { name: "Dashboard" });
});

In the example above, the within helper allows us to scope the test to all the HTML inside the main landmark. This can be really handy and powerful! For example, when we wanted to test the main feed navigation, we could do:

cy.findByRole('navigation', { name: 'View posts by' }).within(() => {
  cy.findByRole('link', { name: 'Relevant' }).as('relevant');
  cy.get('@relevant').should('have.attr', 'aria-current', 'page');
});

We were able to go directly to the feed navigation landmark with ease. This is cool for testing, but it's also been great for building awareness and understanding of how important landmarks and their labels can be for screen reader users. Just like we could skip directly to the "View posts by" navigation in Cypress, screen reader users can skip directly there with their screen reader too! Great news for anyone that wants to get straight to the latest posts 😄

Headings, headings, headings

Headings play an important role in accessibility (I have an older post that digs into this a bit), and they've become really important to our tests too.

One struggle we had with Cypress was that sometimes when we followed a link to a new page, the new page wouldn't have loaded yet before Cypress tried to click some button/check for some behaviour.

For example, we had some test code:

cy.findByRole("main").findByRole("link", { name: "Test article" }).click();
// After clicking the link we can _sometimes_ accidentally get a reference to the 'main' element on the page we just left
cy.findByRole("main").findByRole("button", { name: "Share post" });
// Cypress fails to find the 'Share post' button inside the previous page's `main` element, and the test fails

Since a main existed on both the first and the second page, Cypress would occasionally get a reference to the wrong one 🙈

One way that we prevent this type of flake now is to make sure we look for an element that is unique to the new page, before continuing our test steps. The example above would become:

cy.findByRole("main").findByRole("link", { name: "Test article" }).click();
// Wait for the correct h1 element
cy.findByRole("heading", { name: "Test article", level: 1 });
// Now we can be sure the main below is the right one!
cy.findByRole("main").findByRole("button", { name: "Share post" });

Needless to say, this in turn has made us more aware of pages missing headings, or headings with the wrong level, duplicated, etc. All of which is helping us tighten up our accessibility.

Javascript races 🏁

Aside from these accessibility concerns, we've also become very aware of some side effects of how we handle our Javascript.

We try our best to make sure our pages load quickly, with Javascript asynchronously layered on after the first page load to enhance functionality. In reality, this means it takes some amount of time before some elements on the page are fully initialized and clickable.

For example, when the logged out version of the home feed first becomes visible, the author names don't yet have a click handler to trigger the appearance of the profile preview card - we initialize this functionality after the page first loads.

As a human, it's pretty hard to click the button before it's initialized (largely because the page load is pretty fast 😄), but it's technically possible. Certainly Cypress is speedy enough to do this sometimes!

We've been working around this with a variety of solutions, including using cypress-pipe which lets us retry a click until a certain condition is met. For example, in the case of the feed preview cards we can do:

cy.findByRole('button', { name: 'Admin McAdmin profile details' })
  .pipe(click)
  .should('have.attr', 'aria-expanded', 'true')

This tells Cypress - click this button until aria-expanded is equal to 'true' (meaning the preview card has opened). Don't worry - it won't retry forever - Cypress has a default waiting period for a condition to become true, and pipe will only retry for that long.

It's definitely made us think a lot more about progressive enhancement, and what is and isn't usable from the very first moment of page load.

Want to know more about E2E testing at Forem?

We've been adding to our E2E testing docs as we learn and evolve our approach, including the "gotchas" we've found along the way - feel free to check it out!

I'd love to hear about your experiences with Cypress and Testing Library too!