Note: This post uses the term "mock" as a kind of Test Double that wraps a model so that you can assert how that model was called by the System Under Test (SUT). It's intentionally omitting the details between other Test Doubles such as spies, stubs, dummies, etc.
Let's say that we have a function responsible to "add an index to the database" (fn) that accepts a index (i) which is a primitive Number type. The purpose of the function fn is to run a complex operation before calling another function inside it called "database persist" (db), which receives that same index:
fn(i, db) => db(i)
https://gist.githubusercontent.com/FagnerMartinsBrack/af1451a27c46138bf3d3d890f471c693
Let's say, for the sake of argument, that:
- This is a piece of a legacy code with low test coverage. We need to test the code as much as possible to reduce the chances of breaking how that function currently behaves.
- Because of "1", we can't do Test-First or change the code structure.
- We want to test the "database persist" function with the index input we provide as an argument, not with some arbitrary index partially applied to the "database persist" argument.
Given those constraints, we need to mock the "database persist" function and check if the test will call that function with the value we want:
https://gist.githubusercontent.com/FagnerMartinsBrack/02f94f764b7d703f5bc6c6d784dfe5f3
Everything looks fine, except for the fact we introduced a potential false positive in the test code.
What if inside the function, a code change reassigns the index argument to a value of 5 that is coincidentally the same value we are using for the "fake index" input in the test?
https://gist.githubusercontent.com/FagnerMartinsBrack/d699b5806c8fb6ef1edfd05631bf9c45
In this case, the test will pass because it's not checking if the "database persist" function runs with the index input we provided as an argument, but rather that it's being called with 5, assuming an input of 5.
The index variable, which is assumed to be an invariant throughout the execution of the function to "add an index to the database". In the future, if somebody decides to overwrite that variable, then there will be a non-deterministic test that will keep passing even if the test is not calling the mocked function with the input.
That is not a helpful test.
This problem happens because when we test the arguments of the mocked "database persist" function, we compare the argument value instead of comparing the reference.
Coding so that the test calls the inner mocked function with the primitive input of the parent function can lead to false positives because we are comparing the value, not the reference.
That seems obvious in hindsight, but it can easily pass unseen, even in code review. It's hard to notice that comparing two primitives will create a loophole where a change in the future will make a test pass when it shouldn't.
A solution to this problem, although not a pretty one, is to avoid passing a primitive as the "fake input" in the test. We pass the instance of some dummy object so that when we compare with the "fake input" we can be sure that we are comparing with the actual reference instead of comparing with the value.
It doesn't matter which type the function "add an index to the database" accepts. We can pass an Object Literal just for the sake of holding the reference in the variable so that we can have a proper deterministic test:
https://gist.githubusercontent.com/FagnerMartinsBrack/ea14862fd13c452e9167a90f341eda6e
Passing a dummy Object Literal will be more robust and future-proof. But only if we can duck type the interface* of the original "index" argument to an Object Literal . If the "add an index to the database" function uses the index for arithmetic operations, then passing an Object Literal makes no sense.
* If the meaning of "interface" seems confused in this context, take a look at this post.
If there are arithmetic operations like index + index or index + 1, those cannot be duck typed to an Object Literal, and therefore we will need to use a new instance that provides the same interface of a number, like new Number():
https://gist.githubusercontent.com/FagnerMartinsBrack/ae7af0553c9e90153cca464546f1464f
That will work, because now we are creating a specific instance, and checking against that, instead of against the primitive value.
That will also allow the code to treat the input as a primitive for most use cases throughout the test, so that if the arithmetic operations change in the future, then the test will still pass legitimately.
The instance approach works well when we can substitute the primitive, but it doesn't work when we pass values such as null or undefined, which have no equivalent way to be passed by reference.
Leveraging duck typing to pass by reference instead of passing by value doesn't work when we pass something that doesn't have a reference equivalent that can be duck typed
The example below shows a false positive being introduced when we test the "add index to database" function with an undefined input:
https://gist.githubusercontent.com/FagnerMartinsBrack/eda5b8680776c3605a5a7a2e101395ca
In that case, a solution for the false-positive can be something like property-based generative testing. It will test for the mocking call property using many samples instead of relying on a single one. Unfortunately, that means adding more abstractions into your tests and relying on statistics. Consider the weight of the benefits and the likelihood of your code reassigning the same types that are being generated.
Another solution can be triangulation. Write three tests that use three different values for index, such as undefined, null and 5. If you reassign the variable, at least two tests will fail.
Another one is to design your system in a way you don't need to use mocks.
Mocks can be tricky. Understanding what you want to test and how the language comparison system works is a must to avoid dangerous and subtle traps.
Generative testing and triangulation can be an alternative solution that use statistics to improve the determinism of a mock call. Still, as with most of the things in software development, there are no silver bullets.
This article became a lightning talk: Mocking And False Positives. The slides have other examples and more details.
Thanks for reading. If you have some feedback, reach out to me on Twitter, Facebook or Github.
Top comments (0)