Chrome ❤️ Testing [Testμ 2023]

In this session on Chrome loves Testing, we explored how Chrome’s recent efforts to support testing and automation use cases can benefit developers and testers.

Our speaker for this discussion was Mathias Bynens, a Staff Software Engineer of the Chrome tooling team at Google, with a focus on browser automation. He had extensive experience developing and testing web applications, and his work on Chrome helped make testing and automation more accessible, and reliable. Mathias and his team worked on implementing backend support in the Chrome browser to make things testable.

He has contributed to several open-source projects, including Chrome Driver and Puppeteer, popular tools for automating testing in Chrome. Mathias is passionate about improving the reliability and quality of web applications, and his work on Chrome has helped to make testing and automation more accessible and reliable for developers and testers alike.

If you couldn’t catch all the sessions live, don’t worry! You can access the recordings at your convenience by visiting the LambdaTest YouTube Channel.

Let’s align with the agenda and acquire more knowledge from Mathias’s session.

Agenda

As the session continued, Mathias outlined the session’s agenda and gave a brief introduction.

• Introduction

• Chrome for Testing

• Chrome’s Headless mode: old vs. new

• Q & A Session

Introduction

As the session commenced, Mathias raised a question, stating the real motivation for Chrome testing. He then asked why the Chrome team cared about web testing. He addressed the pain points of testers and developers while providing the solution in further sessions.

Mathias mentioned that the Chrome team understood that setting up an adequate environment for automated browser testing could be challenging. It has been consistently reported as a top pain point for web testing in industry surveys. This presented a great opportunity for them to improve the situation.

He further stated how he and his team worked towards automated browser testing, ensuring that they addressed the pain points of testers and developers.

As the session progressed, he addressed each of the challenges briefly.

Obstacle #1 — Getting a browser binary

Mathias initially emphasized the importance of identifying the fundamental requirement for conducting any form of testing.

Chrome

Mathias mentioned that one of the key requirements for web app testing was a browser binary.
He also mentioned to his audience that If you wanted to test in Chrome, you might have been tempted to install the same Chrome binary on your system. However, this was not an ideal solution.

Chrome drawback

Mathias further discussed that Google intentionally did not provide Chrome downloads for specific versions because they wanted users to always have the latest version for security and feature updates. While this was great for end users, it could be problematic for developers and testers who needed consistent and reproducible results.

To address the Chrome-related challenge, Mathias introduced the Chromium binary as a solution.

Chromium binary

He emphasized that the Chromium binary could assist testers and developers to some extent in reducing the pain points. Unlike Chrome, Chromium binary does not auto-update, making it easier to download specific versions as needed.

Chromium binary drawback

Mathias further discussed that the availability of Chromium binaries was not always reliable across all platforms, which posed a challenge in testing the same version consistently across different platforms. He continued further stating that these binaries were constructed and released independently of the actual Chrome release process, preventing mapping their versions to real user-facing Chrome releases. Furthermore, Chromium differs from Chrome, potentially leading to a slightly varied browsing experience when testing in Chromium.

Mathias further discussed that the availability of Chromium binaries had not always been reliable across all platforms, which had posed a challenge when trying to test the same version across different platforms consistently. He elaborated that these binaries had been constructed and released independently of the actual Chrome release process, preventing mapping their versions to real user-facing Chrome releases. Additionally, he mentioned that Chromium differed from Chrome, which could result in a slightly varied browsing experience when conducting tests in Chromium.

He conducted a reliability assessment for the Chromium binary to provide developers, testers, and his audience with a clearer understanding. This information assisted in making more informed decisions about whether to use it.

Mathias further elaborated that the two had differences that could impact testing. For example, if a web app relied on proprietary video codecs available in Chrome, testing them in Chromium wouldn’t have been possible because Chromium didn’t support them.

In summary, Chromium was not the optimal choice for web app testing.

Chrome for Testing

Mathias progressed by introducing a new solution known as Chrome for Testing. Later in the session, he showed how this new solution could alleviate pain points and demonstrate its efficiency.

Mathias highlighted the advantages of using Chrome for Testing, emphasizing its reliability for testing across various platforms with different versions. He noted that it exhibited minimal differences from Chrome and eliminated the risk of version mismatches. Additionally, he clarified that Chrome for Testing was not accessible to all users but exclusively intended for testing purposes, primarily for experts such as developers and testers.

Downloading Chrome for Testing

Mathias provided a step-by-step guide on downloading Chrome for Testing on any platform using the Puppeteer browser’s command-line utility. It’s available via npm.

Chrome for Testing JSON API endpoint

Mathias explained that users who did not wish to utilize the Puppeteer browsers package had an alternative of crafting their automated scripts for binary downloads. This approach was officially supported, and a JSON API endpoint was made public, providing access to the latest versions for each Chrome release channel.

Chrome for Testing Availability Dashboard

The dashboard was built to summarize the current status conveniently.

Obstacle #2 — Getting a browser driver binary

Mathias stated that while Chrome for Testing addressed several pain points, there were still challenges associated with browser testing. For instance, automating browser testing with WebDriver necessitated a versioned ChromeDriver binary that matched your Chrome binary. In the past, this had entailed a complex version mapping process.

He had instructed developers to follow specific steps to ensure the compatibility of WebDriver with ChromeDriver of any version that developers or testers might need.

The ping address: Mathias stated that the version number could be associated with Chrome, Chromium binary, or Chrome Test, and he emphasized that it wouldn’t matter which one was used when connecting the specific version of WebDriver. The version number had four parts: Major, Minor, Build, and the attachment number.

Remove the attached number: He instructed developers to remove the last attachment number and replace it with the following link: [https://chromedriver.storage.googleapis.com/LATEST_RELEASE.]
This link redirects to another ping address, 114.0.5735.90, where you can find the corresponding versions of WebDriver based on the Chrome browser you intended to connect it to. He explained that the mapping process had been quite challenging to follow.

Furthermore, Mathias provided a solution by stating that, starting with milestone 115, ChromeDriver had been integrated into the Chrome for Testing infrastructure. This guaranteed that for every user-facing Chrome version released, both a Chrome or Chrome for Testing binary and a ChromeDriver binary were available. This eliminated the need for developers and testers to go through any complex version mapping. They had simply been able to select their desired version and download the corresponding binaries. This change allowed anyone to have both Chrome for Testing and WebDriver within the same infrastructure.

Report Availability Dashboard

Mathias reported the availability of ChromeDriver binaries on the Chrome for Testing dashboard. Their JSON API endpoints included relevant information about them.

The installation of ChromeDriver binaries, aligned with the stable version of ChromeDriver milestones, had also been accomplished using the Puppeteer browsers CLI.

Mathias shared valuable resources with his audience where they can gain more information about Chrome for Testing. Continuing with the session, Mathias introduced a Chrome headless test script to be executed as his obstacle 3.

Obstacle #3- Reliable headless automation

Mathias highlighted common challenges when using headless automation. As headless did not support Chrome extensions, this limitation was rooted in technical and historical factors.

The previous headless implementation had been a separate browser version with its codebase, leading to engineering difficulties and the appearance of issues and features not present in the standard Chrome browser.

To overcome this, Mathias addressed a new headless mode that was introduced in Chrome 112. This new mode was the Chrome browser but operated invisibly without displaying platform windows. It resolved the previous limitations and ensured that headless and headful modes were now the same browser.

Old Headless vs New Headless modes

Mathias continued by presenting various use cases for both the old and new headless modes. He further explained that each of these modes had its distinct advantages. The old mode had been lightweight and efficient, making it well-suited for tasks like automated screenshot generation, PDF printing, and web scraping.

The new mode, being the actual Chrome browser, offered more features and authenticity, making it ideal for high-accuracy end-to-end web application testing.

To use the new headless mode, developers simply had to pass the “headless=new” command line flag. Popular automation projects like Puppeteer and Selenium WebDriver had already supported this mode. The old headless mode had been completely removed from the Chrome binary.

In the interim, the old headless mode had remained available as a separate standalone binary called “Chrome Headless Shell” in milestone 118. This allowed developers to choose between the old and new modes based on their specific use cases.

Overall, introducing the new headless mode and the availability of Chrome Headless Shell provided developers and testers with more options and improved testing environments.

Supported Chrome for Testing binaries & Platforms

Mathias had listed the platforms that supported Chrome for Testing binaries. He also highlighted that each Chrome for Testing binary could be used across various platforms, making it an optimal solution for developers. This effectively eliminated the pain points experienced by developers and testers.

Chrome for Testing Ecosystem adoption

Additionally, Mathias showcased their own Puppeteer browsers package, and they had already observed popular ecosystem libraries, frameworks, and tools integrating with Chrome for Testing to simplify user setups while also offering increased reliability and new functionality. Mathias had recommended trying out Chrome for Testing and providing feedback.

It was indeed an interesting session by Mathias. He wrapped up his session by answering a few Q&A’s

Q&A Session

1. How can Chrome browser features enhance testing processes and outcomes? What techniques do you use?

Mathias: Chrome browser features, like Developer Tools and Headless Mode, improved testing by aiding debugging and background testing. Extensions and tools were used to simplify tasks such as accessibility testing and performance analysis. Chrome’s capabilities, coupled with the combination of manual and automated testing techniques, contributed to better testing outcomes.

2. What are the popular Chrome extensions for test automation?

Mathias: One of the achievements the Mathias team worked on was the recorder functionality in Chrome DevTools. We had a recorder panel in DevTools that allowed users to manually execute a user flow on a web page and record the steps. We also enabled the creation of extensions that added their export options to the recorder panel, making it possible to export to third-party services such as Cypress or Selenium. Additionally, we introduced support for replay extensions, allowing users to export and replay actions directly within the browser, right in DevTools.

3. What are the best practices for testing real-time data updates and interactions within web applications running in the Chrome browser?

Mathias: Identify the elements of the web page that are dependable. Knowing the target container where the content is updated enables you to structure your automated tests so that you only verify the content changes when specific actions are performed without getting too concerned about the specific content details.

4. Can you provide information on any limitations of using Chrome for testing?

Mathias: One example of the limitations was using proprietary video codecs in Chrome. While h.264 videos could be played, there were some limitations that hadn’t been addressed at that time. Another limitation pertained to Windows, where the Chrome release process involved A/B testing builds. This occasionally resulted in different Chrome versions on Windows compared to Mac or Linux. At that point, Chrome for Testing did not support the release of all binaries for these A/B builds occurring on Windows.

Got more questions? Please put it on the LambdaTest Community.