Don't target 100% coverage

Don't target 100% coverage... but achieve it anyway!

I recently noticed that a lot of people were advising to reach 100% of code coverage and were presenting it as a primary indicator of code quality. While I agree on the fact that it's an interesting metric, everyone must be aware that having 100% of coverage does not indicate the quality of tests and the way to achieve a good coverage is more important that being able to produce 100% of coverage as an end, it must not be a target in the first place. Let's see why.

Code Coverage

First things first let's briefly explain what is code coverage for those who are not really familiar with the concept.

Code coverage is a technique aiming to determine what parts of your program is being covered by a test. Basically, the code coverage tool instantiates counters and they are incremented once a line of your program is traversed by one of your tests.

Here is a quote from istanbul, one of the most used code coverage tool:

Istanbul instruments your ES5 and ES2015+ JavaScript code with line counters, so that you can track how well your unit-tests exercise your codebase.

Code coverage tools often collect various metrics such as:

• % of statements traversed
• % of branches traversed
• % of functions traversed
• % of lines traversed

Overall, code coverage tools just collect the amount of production code traversed by a test, whether the assertion is relevant or not, which is where the devil resides.

The trap 👹

Let's see a trap example with a simple project using c8 as our code coverage tool (istanbul unfortunately does not natively work with ESM).

You can grab the source code here.

index.js

export function add(a, b) {
  return a + b;
}

Now here is one simple test that we would expect to be truthy:

index.spec.js

import assert from "assert";
import { add } from "./index.js";

it("should return 1", function () {
  assert.equal(add(1, 0), 1);
});

When running the test, we can see that it turns to green and the coverage is automatically generated for us:

As we can see, everything is 100% covered by tests, all the statements, all the functions, all the branches, etc. So everything is fine, so we could just ship the feature right? 🧐

But what if someday someone just changes a little detail, pretty confident that it will work as the unit test still succeeds after the change.

Thanks to Michaël Azerhad who provided me that example while back!

export function add(a, b) {
+  return a - b;
-  return a + b;
}

The test suite still passes and coverage is still showing us 100% of coverage.

As simple as the example may seem, it demonstrates that coverage only checks that a given chunk of code is at least traversed once, but it will never tell you if a test is valuable nor useful in some kind of way. Consequently there is a danger which is that it can make you feel (wrongly) safe about your code, just based on the fact that they are traversed by a test at some point in time.

Don't let "100% coverage" be your primary concern 🖐

Don't get me wrong, I'm not saying that coverage is not useful in itself.

However aiming towards reaching 100% after having written the code will most likely produce false positive tests in a "Test-Last" fashion, just to make statistics look better. I would even say that Code Coverage Driven Development (CCDD) sometimes inherits from the disadvantages of the "Test-Last" approach, that is most of the time either writing irrelevant or even worse useless tests. They both make you feel safe and confident as the console only shows green lights, but it won't avoid you to get in trouble.

Mutation testing, an under-estimated tool 🔍

Because I'm lazy, I'll once again simply quote the definition from the Stryker documentation

Bugs, or mutants, are automatically inserted into your production code. Your tests are run for each mutant. If your tests fail then the mutant is killed. If your tests passed, the mutant survived. The higher the percentage of mutants killed, the more effective your tests are.

Basically, it does something fundamentally different from code coverage which is adding variants to branches of your program. By mutating your code, the mutation testing tool can detect whether your test suite was correctly covering or not the behavior produced by the mutation.

Let's try it on our small example using Stryker.

$ npm run mutation-test

Running the test produces the following output:

As you can see, one mutant survived which is the one changing the Arithmetic Operator which is exactly what we were looking for. Consequently, we can see that it does a step forward by allowing tests to be tested via the mutation of the production code itself. The more your tests kill mutants, the more they tend to be relevant. You could have 100% of coverage by having only 20 to 30% of the mutants killed, which is a big smell. Be careful with code coverage!

You might have guessed it, but mutation testing tools can also provide code coverage reports.

Automatically reaching 100% coverage and 100% of mutants killed using Test-Driven Development 🤹‍♀️

With a simple example we just showed that coverage as such is not enough to ensure tests quality. We also saw that mutation testing helps finding incomplete/missing test suites. What if I told you that a good coverage and a high rate of mutants killed can be a positive side effect of a discipline that does not care about any coverage in the first place?

Test-Driven Development

This article is not dedicated to Test-Driven Development (TDD) is, but I'll try to make a short introduction.

Test-Driven Development is a software development discipline that aims to drive the writing of code required to make a failing test ❌ (prerequisite) pass i.e. turn to GREEN ✅ in very short feedback loop cycles. Test-Driven Development fundamentally helps designing software as it allows to infinitely and safely refactor (at any point in time) the code produced, which is one of the main benefits.

When doing TDD correctly, because each line of code produced must be justified by a failing test in the first place, 100% coverage will be naturally achieved. All the mutants will also be killed if TDD was done correctly, as changing a line of code or one statement will without a doubt break one test, otherwise it means that some code was shamefully produced without having a failing test first!

Not only you'll achieve 100% of code coverage and 100% of killed mutants, but you'll be able to produce a code with a higher level of quality, depending on your refactor skills (because TDD can drive you writing a well designed code but the skillset required to do so has nothing to do with TDD).

Of course Test-Driven Development is not a silver bullet, but it is to date the best way of mixing both code quality with code coverage and the best part is that you'll most likely do it unconsciously.

Wrap up

Don't get me wrong, code coverage is useful in many ways, as it can show how well a codebase is covered by tests at a very high level, for instance having a very low percentage of coverage means that there is not enough tests, so it's good metric to start with.

Nevertheless, having the opposite which is a high percentage of code coverage does not mean that tests are relevant. The best way of achieving both a high percentage of coverage and tests reliability is by not targeting it in the first place, otherwise it might lead people to write more or less (often less) relevant tests just make the percentage a little bit better, the worst part being that it makes everyone feel safer.

By also using mutation testing tools and by mastering disciplines such as Test-Driven Development (or at least Test-First), you can end up achieving 100% coverage and 100% mutants killed by not caring at all (0%)% of all these metrics. That, is the target.

Project sample can be found there: https://github.com/antoine-coulon/dont-target-100-coverage