Setting up mutation testing with stryker and web-test-runner

So, what is mutation testing? Well, it is a type of testing which allows us to evaluate the quality of our tests.

Of course, we could check the code coverage to see if our tests execute all our source code. With that, we could think we are testing all the possibilities and be confident that we don't have any bugs, right?

So let's take a look at this little example:

function compareGreaterThan18(a) {
  return a > 18;
}

Here we can see a simple function, which returns true if the function's parameter is bigger than 18 and false otherwise.

Let's set up our test runner web-test-runner

1. Install web-test-runner:
   

   
     npm i --save-dev @web/test-runner
   
   

2. Install chai:
   

   
     npm i --save-dev @esm-bundle/chai
   
   

3. Create the configuration for wtr (although it can also be executed with just web-test-runner paht/to/*.test.js --node-resolve)

   Just create a web-test-runner.config.mjs file in the root of your project:
   

   
     export default {
       coverage: true,
       files: ['./src/test/*.test.js'],
       nodeResolve: true,
       rootDir: '../../', //
     }
   
   

   rootDir is used to resolve modules in a monorepo, in this case, we need to set it up so Stryker can resolve the modules correctly.

   You can check all the options at https://modern-web.dev/docs/test-runner/cli-and-configuration/

4. Now, we can create our test:
   

   
     import { expect } from '@esm-bundle/chai';
     import { compareGreaterThan18 } from '../compareGreaterThan18.js'
   
     describe('compareGreaterThan18', () => {
       it('should return true if the number is greater than 18', () => {
         expect(compareGreaterThan18(27)).to.be.true;
       });
     });
   
   

5. Execute the test
   

   
     npx wtr
   
   

And with that, we got a 100% of code coverage but are we sure that this test is enough?

No, it is not enough. What happens if someone changes our > inside our code to >=?... Well, the test will still work when it should have failed.

And the same happens if the 18 is changed to another number lower than 27.

In this example, it is easy to see what tests should have added, but it is not always that easy to see what changes in our code could add bugs, and we wouldn't notice it because the tests say everything is ok.

So now, let's see how we can solve this.

Let's set up Stryker Mutator

Stryker is a JavaScript mutation testing framework.

It will modify your code by adding some mutants. For example, in the previous function, it will change the > to >= or it will change it to <.

Then if your tests fail, the mutant is killed, but otherwise, it means the mutant survived, which can indicate that we have not tested everything should have tested.

So let's kill some mutants.

1. Install Stryker
   

   
     npm i --save-dev @stryker-mutator/core
   
   

2. Create the configuration for Stryker

   The file is called stryker.conf.js
   

   
     /**
     * @type {import('@stryker-mutator/api/core').StrykerOptions}
     */
     module.exports = {
       testRunner: 'command',
       files: ['src/*.js', 'src/**/*.test.js', 'package.json', '*.mjs'],
       mutate: ['src/*.js', '!src/**/*.test.js'],
       packageManager: 'npm',
       reporters: ['html', 'clear-text', 'progress'],
     };
   
   

   Here we set up our test runner in this case it will be command as we just want to execute our test command which will be npm test.

   With the files property, you can choose which files should be included in the test runner sandbox and normally you don't need to set it up because by default it uses all the files not ignored by git.

   And then we add the files we want to mutate 'src/*.js' and the ones we don't want to mutate '!src/**/*.test.js' to the array mutate.

   All the options can be checked at https://stryker-mutator.io/docs/stryker/configuration

3. Set your test command to execute wtr
   

   
     "scripts": {
       "test": "wtr"
     },
   
   

4. Modify our web test runner config so it works together with Stryker

   Stryker uses mutation switching to be able to put all the mutants into the code simultaneously, this way it doesn't need to modify your code before running each mutation.

   Then it uses an environment variable to select which mutation is being tested __STRYKER_ACTIVE_MUTANT__.

   With web-test-runner we are running the tests in a browser so we have to inject this variable so the tests can read and use it.

   In our web-test-runner.config.mjs we set the testRunnerHtml property to inject active mutant:
   

   
     function getCurrentMutant() {
       return process.env.__STRYKER_ACTIVE_MUTANT__;
     }
   
     export default {
       coverage: true,
       files: ['./src/test/*.test.js'],
       nodeResolve: true,
       rootDir: '../../',
       testRunnerHtml: testFramework =>
         `<html>
           <body>
             <script>
                 window.__stryker__ = window.__stryker__ || {};
                 window.__stryker__.activeMutant = ${getCurrentMutant()};
                 window.process = {
                     env: {
                         __STRYKER_ACTIVE_MUTANT__: ${getCurrentMutant()},
                     }
                 }
             </script>
             <script type="module" src="${testFramework}"></script>
           </body>
         </html>`,
     }
   
   

   From the version 5 and onwards of Stryker the __STRYKER_ACTIVE_MUTANT__ and activeMutant must be of type String so be sure to put double quotes or single quotes around the expression ${getCurrentMutant()}.
   

   
       window.__stryker__ = window.__stryker__ || {};
       window.__stryker__.activeMutant = '${getCurrentMutant()}'; // Single quotes to be sure it is a string so it works on Stryker version 5
       window.process = {
           env: {
               __STRYKER_ACTIVE_MUTANT__: '${getCurrentMutant()}',  // Single quotes to be sure it is a string so it works on Stryker version 5
           }
       }
   
   

5. Now, we can run our mutation testing
   

   
     npx stryker run    
   
   

   Once it finishes, we will see a report like this one:
   

   In this case, we can see that our test was not able to survive 2 mutants out of 5.

   So now let's kill some mutants!

Let's add some tests to kill the mutants

The first survived mutant is the following one:

-    return a > 18;
+    return true;

The minus symbol indicates what was changed and the plus indicates what was changed.

Here we can see that if our statement was to be changed to always return true, our test would still say that everything is ok which should not be the case and it could be the origin of bugs in the future.

So let's fix it, we have to add a test where we check what happens if a is lower than 18.

it('should return true if the number is greater than 18', () => {
  expect(compareGreaterThan18(14)).to.be.false;
});

With this test, we have killed one mutant and we can kill the one left.

-    return a > 18;
+    return a >= 18;

This mutant is sowing us that we don't check what happens if a is 18 and we don't have any test checking it so we have to add one:

it('should return true if the number is greater than 18', () => {
  expect(compareGreaterThan18(18)).to.be.false;
});

And... congratulations, now we have killed all the mutants!!!!!

Conclusion

With this, we were able to see that code coverage doesn't tell us if our tests are good or bad, instead, we should execute mutation tests as we did with Stryker.

One way to be more confident about our tests, for example, is to check the score calculated by Stryker, the higher the score the more confident we can be about our tests.

And mutation testing can take a lot of time, in the example showed it only takes 3 seconds to execute all the tests, but as your project grows, it will take a lot more.

• Only mutate what you need to mutate, don't mutate your demo folders or your mocks.
• Try to improve the performance of your tests: run tests concurrently, load just what you need to run the tests, stub functions you should not tests, etc

Useful references

• https://modern-web.dev/docs/test-runner/overview/
• https://stryker-mutator.io/

An intro to Mutation Testing - or why coverage sucks

Pedro Rijo ・ Apr 2 '19

#testing #mutationtesting #coverage #java

Who watches the watchmen? Mutation testing in action

Stefano Magni ・ Aug 5 '19

#testing