How to know if your test is flaky?

What are Flaky Tests

Flaky tests are tests that pass sometimes, this can be due to some inconsistency inside it or other reasons.

How to debug a Flaky Test?

A way to find if a test is flaky is to run them N times, if at any point they fail this means that they are flaky.

Instead of repeatedly running tests and manually checking for failures, you can streamline the process by utilizing this script:

function jest-many-times-sequential() {
  i=1
  successes=0
  failures=0
  totalTests=$2
  SUCCESS_CHECKMARK=$(printf '\342\234\224\n' | iconv -f UTF-8)
  CROSS_MARK=$(printf '\342\235\214\n' | iconv -f UTF-8)

  until [ $i -gt $2 ]; do
    if yarn jest $1 --silent; then
      ((successes = successes + 1))
      echo "  $SUCCESS_CHECKMARK tests passed"
    else
      ((failures = failures + 1))
      echo "  $CROSS_MARK tests failed"
    fi
    ((i = i + 1))

  done

  echo "\n
  Ran Tests $totalTests times.\n
  ✅ Succeeded: $successes/$totalTests
  ❌ Failed: $failures/$totalTests
  \n
  "
}

To use this script, just paste the code block into your terminal's configuration file (.zshrc or .bashrc) and then restart your terminal.

Then you can run the script as follows:

$ jest-many-times-sequential <path-of-your-test> <times-to-run>

By running the tests N times sequentially, the script will automatically track the number of successful and failed runs, providing you with a final tally at the end.

Here is an example:

Now with this script we can validate if our tests are flaky or not.

Woovi is a Startup that enables shoppers to pay as they like. To make this possible, Woovi provides instant payment solutions for merchants to accept orders.

If you want to work with us, we are hiring!

Photo generated by DALL-E