Tips to avoid unstable tests in Selenium

Selenium tests are hard to maintain. They can get flaky and present intermittent failures as their codebase grows.

I will list a few things that I've seen as effective to make Selenium tests stable, running a suite of ~10K tests.

TL;DR;

• Test code must be treated as seriously as development code (do code review, use best practices);
• Track the problems creating bugs for failures;
• Use Page Objects;
• Understand how Selenium wait works (implicit, explicit) and to use WebDriverWait and ExpectedConditions classes.

1. Treat test code in the same way you treat development code

Test code is not different from development code. They act in different areas of the software, but they should be treated with the same respect.

The more effort is put in having good code for automated tests, fewer problems are found when maintaining the test suites.

Do code review

Code for automated tests MUST be reviewed. They have to be treated with respect, and the reviewer has to be as strict as he or she is with the code from the developers.

E2E code is widely seen as expensive and unstable. Code review helps to organize the code in reusable components, enforce using better selectors (css, id, name or even xpath), check code semantics, ensure it has good logging, verify that it is well-documented and that errors are handled properly (etc, etc, etc).

When code review is taking seriously for E2E tests (or any automated test), the consistency of the codebase enforces standards that result in more stability in tests execution, reducing the issues seen when running E2E tests frequently, or after new releases.

Request for best practices

Page Objects have been around for a long time. They are a solid pattern to be used. However, it is not the only thing to be used.

Create random data to enter in fields for tests. There are many libraries that can help (e.g. DataFaker, jFairy). Do not use static values.

Use the Builder pattern to create instances of entities.

Inject sessions to skip repeatedly log-in operations.

Keep specifications small. Try to set up a threshold for the overall time it takes to run (I use 5 minutes as a reference).

Use DSL, if convenient.

Choose a good a testing library to perform assertions (JUnit, TestNG, ScalaTest).

Run tests over a CI server (Jenkins, TeamCity).

Scale the execution using Selenium Grid - it is easy with Docker Selenium.

Use Page Objects

As said before, Page Objects are already mature enough to be called a pattern.

They can be used not only to map pages, but also components.

Pages

Page is a UI representation that must be reusable in the codebase. Different test cases can access the same page and its code must be shared (e.g. the system's landing page).

The page class must contain fields and methods to interact with the web elements that are present on the UI.

Components

A page is composed of many different web components. They can be seen as inputs, dropdowns, custom elements and so on.

A component can be reusable along different pages. They should be wrapped in a specific class that can be instantiated in different pages, similar with how the front-end codebase declare them.

2. Create bugs for all issues

All tests that fail and need to be fixed need an issue in the issue tracker system. As much as any intermittent failure needs to have one.

When creating issues for all failures seen, issues in automated tests can be tracked. When issues can be tracked, they can be prioritized and, if they can be prioritized, they can be fixed in an order that helps the suite of tests to become more stable.

Remember:

• if a test case is failing consistently (it can even be reproduced manually): create a ticket to fix it.
• if a test case is failing intermittently: create a ticket to investigate and fix it.

There must be no fear for tickets. They serve to help.

3. Implement custom waiting

Page load is probably the thing that mostly impacts in Selenium tests performance. They are hard to predict and sometimes lead to intermittent failures.

If a screenshot is taken by the time that a Selenium test fails, it reveals commonly that there is something still loading on the page (the page itself or one of its components).

It is extremely important to understand how to implement a mechanism for Selenium to apply some waiting time.

Implicit Wait

Implicit wait is defined by values that are set up in the WebDriver instance.

// Specifies the amount of time the driver should wait when searching for an element if it is not immediately present.
webDriver.manage.timeouts.implicitlyWait(3) 
// Sets the amount of time to wait for a page load to complete before throwing an error
webDriver.manage.timeouts.pageLoadTimeout(6)

These are default values set in the WebDriver that will be used between starting the web browser (webDriver.get(url)) and closing it (webDriver.quit()).

Explicit Wait

Implicit wait is not always effective. Pages and their elements can take different time to load.

An exception like NoSuchElementException is commonly thrown when Selenium cannot find the element to interact with on the page.

If the element is excepted to be on the page, it is possible to use other strategies to wait for it, for longer than the Implicit Wait set into the WebDriver.

This can be achieved combining the WebDriverWait and the ExpectedConditions and is called Explicit Wait.

WebDriverWait and ExpectedConditions

The API for the ExpectedConditions is rich and very clear. It is used internally inside an instance of the WebDriverWait class, passing by parameter also the time that it should wait for the condition.

Below some self-explanatory examples:

// Instantiate the `WebDriverWait` with `15` seconds as the limt time to wait for a condition
WebDriverWait = bew WebDriverWait(driver, 15)

// Find an element on the page
WebElement element = webDriver.findElement(By.id('main'));

// Wait for an element to be stale
seleniumWait.until(ExpectedConditions.stalenessOf(element));

// Wait for an element to be visible
seleniumWait.until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector(".button"));

// Wait for an element to be invisible
seleniumWait.until(ExpectedConditions.invisibilityOfElementLocated(element));

// Combined condition: all 3 elements must be clickable
seleniumWait.until(
    ExpectedConditions.and(             
        ExpectedConditions.elementToBeClickable(By.id("a")),
        ExpectedConditions.elementToBeClickable(By.id("b)),
        ExpectedConditions.elementToBeClickable(By.id("c"))
    )
)

// Wait for 2 elements present on the DOM
seleniumWait.until(ExpectedConditions.numberOfElementsToBeMoreThan(By.cssSelector(".button"), 2))

The API for the ExpectedConditions class is rich and easy to understand. There are many methods that can help the conditions that could help to add stability to tests that rely on elements that take longer than expected to be ready to interact with.

4. It's not all

Surely this is an endless discussion, and many other things could be added to the article above that contribute to having stable tests implemented with Selenium.