Property-based testing for JavaScript developers

Special thanks to Nicolas Dubien for sharing his generative JavaScript testing wisdom with me before I began writing this guide.

All experienced frontend developers know one thing to be true: Users are unpredictable. No matter how much user research you conduct or how thick the font-weight is on your input label, you can never be certain how users will interact with your product. That’s why, as the creators of the interface, we put in constraints. And to ensure that those constraints work properly, we write tests.

But there's a problem with traditional unit and integration tests.

They require us to manually think of and write every scenario that our tests will cover. Not only does this take a lot of time, but it also limits the test coverage to our imaginations. Whereas users, as we know, are unpredictable. So we need a way to test our software to withstand an unlimited number of potential user flows.

That’s where property-based testing comes in.

Within this guide, we’ll explain the must-knows of property-based testing in JavaScript. We'll walk through practical examples and you'll write your first test using the fast-check framework. Finally, we'll touch on what other property-based testing frameworks are out there.

What's in this guide

• Property-based testing in JavaScript: What and why
• Differences between property-based and example-based tests
• What your existing test suite probably looks like (and is missing)
• Choosing which properties to test for
• Writing your first property-based test with fast-check
• Available property-based testing frameworks
• Conclusion

⚠️ Prerequisites:

• A solid understanding of what unit tests are.
• Familiarity with Jest or another JavaScript testing framework.
• (Optional) NPM or Yarn installed if you want to follow along in your IDE.

🐍 Prefer Python? A similar guide is available to help you learn property-based testing in Python instead.

💻 References:

We've created a GitHub repository to accompany this guide. This repository includes all of the featured tests with instructions for how to execute them. It also provides more resources for learning property-based testing.

Property-based testing in JavaScript: What and why

Software testing as we know it today requires a lot of time and imagination. When you're writing traditional example-based tests, you're stuck trying to manually reproduce every action that a user might make.

Property-based testing is a different approach to writing tests designed to accomplish more in less time. This is because instead of manually creating the exact values to be tested, it's done automatically by the framework you're using. That way, you can run hundreds or even thousands of test cases in the same amount of time it takes you to write one expect statement.

As the developer writing the tests, what you have to do is:

• Specify what type of values the framework should generate (i.e. integers or strings).
• Assert those values on guarantees (or properties) that are true regardless of the exact value.

We'll cover how to choose which properties to test for later in this guide. But before going any further, let's talk about why you would want to integrate property-based testing into your workflow.

Nicolas Dubien, the creator of the fast-check framework we're exploring in this guide, wrote a post outlining the primary benefits of property-based testing.

To summarize his words, property-based testing enables developers to:

• Cover the entire scope of possible inputs: Unless you specifically tell it to, property-based testing frameworks don't restrict the generated values. As a result, they test for the full spectrum of possible inputs.
• Shrink the input when tests fail: Shrinking is a fundamental part of property-based testing. Each time a test fails, the framework will continue to reduce the input (i.e. removing characters in a string) to pinpoint the exact cause of the failure.
• Reproduce and replay test runs: Whenever a test case is executed, a seed is created. This allows you to replay the test with the same values and reproduce the failing case.

In this guide, we'll focus on that first benefit: Covering the entire scope of possible inputs.

Differences between property-based and example-based tests

Even with the limitations mentioned, traditional example-based tests are likely to remain the norm in software testing. And that's ok because property-based tests aren't meant to replace example-based ones. These two test types can, and very likely will, co-exist in the same codebase.

While they may be based on different concepts, property-based and example-based tests have many similarities. This becomes evident when you do a side-by-side comparison of the steps necessary to write a given test:

At its core, property-based testing is meant to provide an additional layer of confidence to your existing test suite and maybe reduce the number of boilerplate tests. So if you're looking to try out property-based testing but don't want to rewrite your entire test suite, don't worry.

What your existing test suite probably looks like (and is missing)

Because property-based tests are meant to fill the coverage gaps missed by traditional testing, it's important to understand how these example-based tests work and their downfalls.

Let's start with a definition: Example-based testing is when you test for a given argument and expect to get a known return value. This return value is known because you provided the exact value to the assertion. So when you run the function or test system, it then asserts the actual result against that return value you designated.

Enough theory, let's write a test.

Imagine you have an input where users write in a number indicating an item's price. This input, however, is type="text" rather than type="number" (trust me, it happens, I've seen it). So you need to create a function (getNumber) that converts the input string into a number using JavaScript's built-in Number() function.

It might look like this:

// getNumber.test.js
const getNumber = inputString => {
  const numberFromInputString = Number(inputString)
  return numberFromInputString
}

Now that you have your getNumber function, let's test it.

To test this using example-based testing, you need to provide the test function with manually created input and return values that you know will pass. For example, the string "35" should return the number 35 after passing through your getNumber function.

// getNumber.test.js
test("turns input string into a number", () => {
  expect(getNumber("35")).toBe(35)
  expect(getNumber("59.99")).toBe(59.99)
})

Note: To run this test on your IDE, you'll need to have Jest installed and configured.

And with that, you have a passing example-based test 🎉

Recognizing the limitations of example-based testing

There are many situations where an example-based test like this would work well and be enough to cover what you need.

But there can be downsides.

When you have to create every test case yourself, you're only able to test as many cases as you're willing to write. The less you write, the more likely it is that your tests will miss catching bugs in your code.

To show how this could be a problem, let's revisit your test for the getNumber function. It has two of the most common ways to write a price value (whole number and with a decimal):

// getNumber.test.js
test("turns input string into a number", () => {
  expect(getNumber("35")).toBe(35)
  expect(getNumber("59.99")).toBe(59.99)
})

Both of these test cases pass. So if you only tested these two values, you might believe that the getNumber function always returns the desired result.

That's not necessarily the case though. For instance, let's say your website with this price input also operates in Germany, where the meaning of commas and decimals in numbers are switched (i.e. $400,456.50 in English would be $400.456,50 in German).

So you add a third test case to address this:

// getNumber.test.js
test("turns input string into a number", () => {
  expect(getNumber("35")).toBe(35)
  expect(getNumber("59.99")).toBe(59.99)
  // Add a new test case:
  expect(getNumber("19,95")).toBe(19.95)
})

But when you run the test... you hit a Not-A-Number error:

expect(received).toBe(expected) // Object.is equality

Expected: 19.95
Received: NaN

Turns out the getNumber function doesn't work as expected when the input string contains a value or specific characters that Number() doesn't recognize. The same error occurs with inputs like twenty or $50. Maybe you already knew that, but maybe you would've never known that without a specific test case.

🐛🚨 This is one example of how property-based testing can be used to find bugs in your software. Once you realize that any string with a character that Number() doesn't recognize will return NaN - you might reconsider how you built that input. Adding the attribute type="number" to the input restricts the possible values that users can enter and, hopefully, helps reduce bugs.

Choosing which properties to test for

Issues like the one faced with the input type also help you write your property-based tests because then it's more clear what the property you're testing for actually is.

Let's dig into this. In property-based testing, a property is an aspect of the function being tested that's always true, regardless of the exact input.

If you look at the getNumber function from earlier, one property you'd test would be the string that is passed to getNumber. Regardless of whether that input value ends up being "59.99", "twenty", or "$50" - it will always be a string.

Some other examples of properties:

• List length when testing the sort() method on an array. The length of the sorted list should always be the same as the original list, regardless of the specific list items.
• Date when testing a method for the Date object like toDateString(). No matter the specifics entered, it will always be a date.

Writing your first property-based test with fast-check

To put property-based testing into practice, let's create an example test using fast-check, a JavaScript framework for generative test cases.

Let's use the getNumber function from earlier. As a reminder, here's what that looked like:

// getNumber.test.js
const getNumber = inputString => {
  const numberFromInputString = Number(inputString)
  return numberFromInputString
}

Now let's write a property-based test using fast-check. To limit the scope, you'll only generate input strings with floating-point numbers because values with decimals are more common in prices.

Structuring your tests

When getting started with fast-check, you first have to set up the base structure of your tests.

Initially, it'll look identical to any other Jest test. It starts with the test global method and its two arguments: A string for describing the test suite and a callback function for wrapping the actual test.

test("turns an input string into a number", () => {
  // Your property-based test will go here!
})

Introducing fast-check

Next, you'll import the framework and introduce your first fast-check function: assert. This function executes the test and accepts two arguments: The property that you're testing and any optional parameters. In this case, you'll use the property function to declare the property.

const fc = require("fast-check")

test("turns an input string into a number", () => {
  fc.assert(
    fc.property(/* Your specific property and expect statement will go here */)
  )
})

Testing your chosen properties

Finally, you'll add the details of the specific values you want to generate. There's an entire list of built-in arbitraries (aka generated datatypes) provided by fast-check. As mentioned previously, this test will cover input strings with floating-point numbers. There are multiple arbitraries for generating floating-point numbers, but this test will use float.

This float arbitrary will be passed as the first argument of the property function, followed by a callback wrapping the expect statement and any other logic necessary for executing the test.

In this test, testFloat represents each floating-point number generated by fast-check and it's then passed as an argument to the callback. The expect statement indicates that when you pass the testFloat as a string to your getNumber function, you expect it to return the same testFloat value as a number.

test("turns an input string into a number", () => {
  fc.assert(
    fc.property(fc.float(), testFloat => {
      expect(getNumber(`${testFloat}`)).toBe(testFloat)
    })
  )
})

And there you have it, your first property-based test 🎉

Examining the generated values

By default, the property check will be run against 100 generated inputs. For many arbitraries, you can also set a minimum or maximum number of generated inputs. At first, running hundreds of test cases might feel excessive - but these numbers are reasonable (and even considered low) in the property-based testing realm.

Going back to the example test, you can peek at the generated input values using fast-check's sample function. This function takes in an arbitrary or property and the number of values to extract. It then constructs an array containing the values that would be generated in your test.

fc.sample(fc.float(), 10)

If you wrap the previous function in a console.log() statement, you'll get something like this:

7.1525
1.3996
0.8122
0.0004
3.5762
0
5.9604
9.5367
0.1504
8.3446

Note: Your numbers will likely be different - and that's ok. You also might not want to log a bunch of random numbers in your terminal - and that's ok too. You can take our word for it.

Available property-based testing frameworks

We opted to use the fast-check framework for this guide, but there are many other options out there to help you write property-based tests in a variety of programming languages.

JavaScript

• JSVerify
• node-quickcheck

Other languages

• Hypothesis: Python (used in our Python guide)
• FsCheck: .NET
• jqwik: Java
• PropCheck: Elixir
• Proptest: Rust
• PropEr: Erlang
• RapidCheck: C++
• QuickCheck: Haskell
• QuickCheck ported to Rust: Rust
• ScalaCheck: Scala

Conclusion

While it won't replace example-based tests, property-based testing can supply additional coverage where traditional tests fall short. One of the benefits of property-based testing is that it helps cover the entire scope of possible inputs for any given function. We explored that benefit throughout this guide by creating a getNumber function and writing a test that uses a generative floating-point number property.

This guide wasn't intended to be a series, but the possibility of future guides about shrinking, replaying tests, property-based testing in TypeScript, or our favorite fast-check features emerged during our research. If that sounds interesting to you, comment below or tweet at us!

At Meeshkan, we're working to improve how people test their products and APIs. So if you made it to the end of this guide and have thoughts, we want to hear from you. Reach out on Gitter or open an issue on GitHub to let us know what you think.