How good is tagging?

Rhetorical question; Tagging is great; my favourite organisation system, in fact. Test systems with tagging allow you the flexibility to organize your tests according to what code they're testing, while running tests according to what kind of test you're after.

Tagging & Cypress

Unfortunately, Cypress lacks native support for tagging, and this makes me very sad face emoji.

Fortunately, with saucectl, we can retrofit tagging functionality onto our Cypress tests, whether you're running locally in Docker mode, or remotely against Sauce Labs' cloud.

Wait, what's saucectl?

Let's go to the README!

The saucectl command line interface orchestrates the tests in your framework, providing rich parallelization, test history filtering, and analytics in Sauce Labs.

Basically, saucectl is an amplifier for your tests. It drags your Cypress, Playwright, TestCafe or Puppeteer kicking and screaming onto the Sauce Labs cloud, giving you delicious Sauce features (like analytics) along with all the goodies your framework has to offer. Saucectl can also run your tests with Docker, giving you a quick and reliable test environment. It's a great solution for using Cypress with CI.

Our Approach

In config.yml (the config file for saucectl), you can specify individual suites see here which let you provide individual config options to Cypress.

By passing the testFiles option to Cypress, you can control which individual files it runs. We're going to combine these two, to give us individually named test suites, each of which runs tests with a specific set of tags.

Can't you do that with Cypress alone?

Yes, technically, but I prefer this approach for... let's call them ideological reasons.

While you could have individual config files, and call cypress run --config-file somesuite.json for each one, it's messy. If you make changes to one suite file, you have to make it across all of them, which is easy to forget.

With this solution, all your Cypress config stays as is, and only the tests being executed are changed. Even better, control of which tests run is maintained in the infrastructure configuration, and infrastructure is likely to have the most significant impact over what 'flavours' of test you want to run.

How we're implementing tagging

This solution makes use of globbing, a way of selecting a group of files. It's pretty common; Common enough to have a Wikipedia page, and you've likely seen at least one glob before.

We're going to alter our tests so each test suite is configured to run a set of globs corresponding to our tagged tests.

Example

Suite Files

Say this is our test suite. Each file contains tests that should only run on a specific browser (or combination of browsers). We've named them so each filename includes the relevant tags, separated by underscores, like _chromeOnly or _ieOnly. These could just as easily be _ui or _FrenchLanguage or anything you want to tag by.

specs/
  ui/
    login_chromeOnly_spec.js
    login_firefoxOnly_spec.js
  accounting/
    jsMathBugFix_ieOnly_spec.js
    chromiumCantCount_edgeOnly_chromeOnly_spec.js

Config.yml

Here's the suites component of our config.yml. Each suite has a unique glob for testFiles which finds all tests in your specs folder which include the respective tag/s.

  suites:
    - name: "Where There's Smoke There's"
      browser: "firefox"
      platformName: "Windows 10"
      config:
        testFiles: ["specs/**/*firefoxOnly*_spec.js"]

    - name: "Cream on"
      browser: "chrome"
      platformName: "Windows 10"
      config:
        testFiles: ["specs/**/*chromeOnly*_spec.js"]

    - name: "Livin' on the"
      browser: "edge"
      platformName: "Windows 10"
      config:
        testFiles: ["specs/**/*edgeOnly*_spec.js", "specs/**/*ieOnly*_spec.js"]

How it works

Each glob starts with specs/**/*. This glob means "start in the specs folder (specs/), then look in any subfolder(**), for any characters at all(*) followed by an underscore(``)".

Let's look at how the first glob continues. After specs/**/* we have firefoxOnly*_spec.js. This means "find anything containing the phrase firefoxOnly, followed by zero or more of any character(*), followed by `_spec.js". For our tests, that only matches specs/ui/login_firefoxOnly_spec.js`.

The second glob is specs/**/*chromeOnly*_spec.js, so it matches every file in every subfolder of specs which contains chromeOnly, followed by zero or more of any character, followed by _spec.js. In our case, that's specs/ui/login_chromeOnly_spec.js AND specs/accounting/chromiumCantCount_edgeOnly_chromeOnly_spec.js

Did you notice?

The testFiles parameter doesn't take a glob directly; it takes an array of globs. That lets us pass in multiple entries, which is how suite three works. We're able to pass in two entries, one for any test tagged edgeOnly and one for any tagged ieOnly. It will match any file in either of those blobs; in this case specs/accounting/jsMathBugFix_ieOnly_spec.js and chromiumCantCount_edgeOnly_chromeOnly_spec.js.

Additionally, test files can have multiple tags. This is really useful when you want to run some tests in multiple scenarios; Say you have a limited set of functional tests that are also used to ensure deploys go well; You could tag them _functional_postDeploy_spec.js and they'd run when using either the _functional tag or the _postDeploy one.

And that's it!

A small change to how you name and select tests, and you're left with a more flexible testing system. And of course, you can still run a default suite with no tags, and make sure you're running everything.

Happy Testing!

Credits

Photo by Angèle Kamp
on Unsplash