Integrate tests with Buildkite Test Analytics

Buildkite introduced a new feature called Test Analytics 📈 this month and in my attempt to rank high on SEO on this subject, you're in the right article on how I did it (considering you already have access to Buildkite).

Just a bit of context (which no one cares), I just joined a scale-up where we started tracking stuff and understandably, it includes the quality of our delivery lifecycle (or else, what's the point of my existence 😂). One of the metrics I personally wanted to check is how predictable and trust-worthy our tests are. The tendency is to ignore tests (especially on the Slack alerts where it becomes normal) when there is pressure to deliver and the amount of failing tests exceed the succeeding ones. That is just unacceptable for me as a Quality engineer and something that should be closely monitored.

"You can't improve what you don't measure." — Peter Drucker

For our jest frontned unit tests, there is a readily-available integrator (which makes life so easy!) where you just simply need to install (and obviously, include in your source code repo) the npm package called buildkite-test-collector and include the reporter in your config file (e.g. jest.config.js):

reporters: [
  'default',
  'buildkite-test-collector/jest/reporter'
],
testLocationInResults: true

For our cypress end-to-end tests, it is a bit trickier as it involves a couple more steps. You need to use the default junit reporter and include the reporter in your config file (e.g. cypress.json or cypress.config.ts for Cypress 10 and above):

  "reporter": "junit",
  "reporterOptions": {
    "mochaFile": "results/junit-[hash].xml",
    "toConsole": true
  }

Once done, you need to go to your test or build pipeline (or wherever you run your tests) > Build steps (or pipeline.yml), and import the JUnit XML file. As lazy as I am (some call it smart 😎), I used the Test collector plugin in this instance.

steps:
  - label: '🧪 Run cypress tests'
    commands: 'npx cypress run --record --parallel --ci-build-id $BUILDKITE_BUILD_NUMBER'
    plugins: # pass details to Test Analytics
      - test-collector#v1.0.0:
          files: 'results/junit-*.xml'
          format: 'junit'
    artifact_paths:
      - 'results/**/*'

Once the configuration is done, include the BUILDKITE_ANALYTICS_TOKEN value on your pipeline's Build Steps or pipeline.yml

Once the tokens are set, run your pipelines and then the magic happens! 🪄

Note: Unsure why the test summary is not included in the main test suite page for the unit tests (maybe the Buildkite Gods can help us and I will update this write-up).

Click a specific test suite and the "devil (if heaps of failing tests) is in the details" 👿

This is good alternative to Cypress dashboards too but free of charge if you're already subscribed to Buildkite.