How to write fewer tests but find more bugs

How in general should I know what to test?

Whenever we're deciding how to test a component, the main challenge is choosing which tests to write. That's because even a simple function like add(a: number, b: number) has a potentially infinite number of input values it can receive. And since we have limited time and budget we can't do them all. Thus we need to be able to choose a small number of inputs, out of all the possible inputs, that will reveal as many bugs as possible.

To solve this issue, I've been using an approach that combines Input Space Partitioning and Whitebox testing.

Input space partitioning

To put it simply, the idea behind Input Space Partitioning is that by analyzing the desired outputs of a piece of code, we can group its inputs such that if the code works for an input of a group, it will also work for any input of that same group. Therefore, we only need to write one test for each group.

Note that inputs include everything that affects the behavior of a component (e.g. props, user action, API response values, etc...), and outputs everything it produces (e.g. rendered elements, API requests, values persisted to storage, etc...).

Take as an example a FizzBuzz inspired React component. The component should allow users to type numbers. When given a number that's a multiple of 3 the component should show Fizz, a number multiple of 5 should show Buzz, a number multiple of 3 and 5 should show FizzBuzz, and a number that's multiple of neither 3 or 5 shows the given number.

Following the logic of Input Space Partitioning, the FizzBuzz input domain can be split into four different categories which are represented by the left column of the table above. This means that we only need to write four tests, one for each of the input categories.

WhiteBox testing

You might be wondering how can we be sure, just by looking at the description of the behavior of the FizzBuzz component, that we've chosen the minimal amount of tests that will reveal as many bugs as possible. The answer is we can't. And that's why we also rely on Whitebox testing.

Whitebox testing, in this context, means we'll use the knowledge of how a component is implemented to decide which tests to write. By looking at the implementation, we can have a better idea of what bugs we might have and thus allow us to choose tests more cost-effectively.

Example 1 - Implementation matches the Input Space Partitioning analysis

If the FizzBuzz code is written as follows, then for each input category, we only need to write one test assertion.

function FizzBuzz() {
  const [value, setValue] = useState(1)

  function fizzBuzz(number: number) {
    if (number % 3 === 0 && number % 5 === 0) return "FizzBuzz"
    if (number % 3 === 0) return "Fizz"
    if (number % 5 === 0) return "Buzz"
    return number
  }

  return (
    <>
      <label htmlFor="fizzBuzz">Enter a FizzBuzz number:</label>
      <input
        type="number"
        id="fizzBuzz"
        name="fizzBuzz"
        value={value}
        onChange={e => setValue(Number(e.target.value))}
      />
      <p>{fizzBuzz(value)}</p>
    </>
  )
}

The corresponding tests for this implementation would be as follows:

test.each`
  number  | result        | description
  ${"15"} | ${"FizzBuzz"} | ${"Multiples of 3 and 5"}
  ${"6"}  | ${"Fizz"}     | ${"Multiples of 3 but not 5"}
  ${"10"} | ${"Buzz"}     | ${"Multiples of 5 but not 3"}
  ${"7"}  | ${"7"}        | ${"Multiples of neither 3 or 5"}
`("$description - $number", ({ number, result }) => {
  render(<FizzBuzz />)
  userEvent.type(screen.getByLabelText("Enter a FizzBuzz number:"), number)
  expect(screen.getByText(result)).toBeVisible()
})

We don't need to write more than one assertion per input domain because with just one assertion we cover all the input domains we determined in the Input Space Analysis, and we cover all the relevant code branches.

Example 2 - Implementation has more branches than Input Partitions

function FizzBuzz() {
  const [value, setValue] = useState(1)

  function fizzBuzz(number: number) {
    if (number === 1) return "1"
    if (number === 2) return "2"
    if (number % 3 === 0 && number % 5 === 0) return "FizzBuzz"
    if (number % 3 === 0) return "Fizz"
    if (number % 5 === 0) return "Buzz"
    return number
  }

  return // rest as it was...
}

If we're given an implementation like the one above, then one test assertion per input domain won't be enough, since the first two branches of the fizzBuzz function won't be covered. So we'll need to adjust the test assertions so we cover everything in the Multiples of neither 3 or 5 partition.

test.each`
  number  | result        | description
  ${"15"} | ${"FizzBuzz"} | ${"Multiples of 3 and 5"}
  ${"6"}  | ${"Fizz"}     | ${"Multiples of 3 but not 5"}
  ${"10"} | ${"Buzz"}     | ${"Multiples of 5 but not 3"}
  ${"7"}  | ${"7"}        | ${"Multiples of neither 3 or 5"}
  ${"1"}  | ${"1"}        | ${"Multiples of neither 3 or 5 - special case 1"}
  ${"2"}  | ${"2"}        | ${"Multiples of neither 3 or 5 - special case 2"}
`("$description - $number", ({ number, result }) => {
  render(<FizzBuzz />)
  userEvent.type(screen.getByLabelText("Enter a FizzBuzz number:"), number)
  expect(screen.getByText(result)).toBeVisible()
})

One might argue that those first two assertions are simple enough that they're obviously correct and thus not worth testing. That's a fair observation and one of the advantages of this way of testing is exactly that we can take the implementation into account to write fewer tests. I'd still argue that it's a good principle to have every bit of code run at least once during tests, but I wouldn't reject a PR due to this.

In case you're wondering, changing fizzBuzz so we only need one assertion per test is an option. So if you're ever in a situation like this, take the opportunity and try to simplify the code.

Example 3 - Implementation uses a production-grade library

Imagine this implementation that uses a library underneath that's been battle-tested. Which tests should we write for it?

function FizzBuzz() {
  const [value, setValue] = useState(1)

  function fizzBuzz(number: number) {
    return battleTestedFizzBuzz(number)
  }

  return // rest as it was...
}

I'd argue we only need one. Since the underlying library gives us confidence that the FizzBuzz logic works as expected, and the React-specific code is straightforward, just one test to see that the code runs should be enough.

test("Runs as expected", () => {
  render(<FizzBuzz />)
  userEvent.type(screen.getByLabelText("Enter a FizzBuzz number:"), "15")
  expect(screen.getByText("FizzBuzz")).toBeVisible()
})

Example 4 - Really complex implementation

To finish these examples, take a look at the project FizzBuzzEnterpriseEdition. Imagine that somehow the React component communicated with a running instance of that project to know what it should show the user based on its input. What tests would you write for it?

My answer is that I don't know. Aside from picking one test assertion per partition determined in the Input Space Analysis, I have no idea what other inputs to pick. The code is so complex that it hides the bugs it might have.

All of these examples give us an interesting insight. The harder the code is to understand, the more test we'll have to write to be confident it works. Therefore, having a clear logic for what we're implementing is essential to enable effective testing.

Put it into action

If you were not familiar with any of the ideas in this article, this can be a lot to process. So here's a summary of how you can put these ideas into practice next time you have to test a component.

1. Start by explicitly defining the behavior of the component.
2. Make sure that for every possible input you know what the output should be.
3. Partition the inputs based on the characteristics of the produced outputs.
4. Look at the implementation of the component.
5. Verify if one test per input partition is enough or too much.
6. Write the tests.

Would you like to see a more complex example?

I wrote a follow-up article to this one where I go over a more complex component and test it using the methodology described in this article. It is available for subscribers of my newsletter. So if you'd like to see it, be sure to subscribe here.