In CI, Run Fast and Fundamental Tests First

While it’s great to keep your CI/CD process fast, on many occasions you don’t even need to run all tests to get the feedback you need.

Unit tests run the fastest, because they test a very small piece of code in complete isolation from the rest of the system. For example, they usually don’t touch the database.

Unit tests deal with fundamental business logic, and are the most numerous, as is commonly depicted in the Test Pyramid diagram:

A failure in unit tests is a sign of a fundamental problem, which makes running the remaining high-level and long-running tests irrelevant.

For these reasons, projects with test suites that run for anything longer than a minute should prioritize unit tests in the continuous integration (CI) process.

How to Use Pipelines to Run Fast Tests First

A classic CI setup is to split all tests in parallel jobs:

In this approach, the total CI build time is determined by the duration of the slowest parallel job. For example:

• If some unit tests have failed in less than a minute,
• But one of the jobs with UI tests runs for 6 minutes,
• Then you need to wait for 6 minutes to receive a report about unit test failures.

The unit tests that fail typically make some higher-level tests fail as well. However, in this case the results of higher-level tests are irrelevant and running them is a waste of time and resources.

A better approach is to use customizable CI/CD pipelines. This way you can configure sequential blocks of single or parallel jobs. As soon as one block fails, the pipeline stops.

This strategy allows developers to get feedback on trivial errors in seconds. It also encourages all team members to understand the performance impact of individual tests as the code base grows.

More Tips for Fast Feedback

There are additional tactics that you can use with your CI system to get fast feedback:

• Conditional stage execution lets you defer running certain parts of your build for the right moment. For example, you can configure your CI to run a subset of end-to-end tests only if one of the related components was changed.
• A fail-fast strategy gives you instant feedback when a job fails. CI stops all currently running jobs in the pipeline as soon as one of the jobs has failed. This approach is particularly useful when running parallel jobs with variable duration.
• Automatic cancelation of queued builds can help in situations when you push some changes, only to realize that you have missed something small, so you push a new revision immediately, but then need to wait for twice as long for feedback. With this approach you get feedback on revisions that matter while skipping all the intermediate ones.

The same principle applies to stages in the delivery pipeline. For example, if you’re running smoke tests after deployment, you want to detect issues as quickly as possible.

What is your approach to getting fast feedback? Let me know in the comments. Happy building!