Working with non-user asynchronous events in model based tests with XState

I recently blogged about my about experience in developing model based tests using state machines with the help of XState and it's companion library @xstate/test. I the article I talk about intercepting network requests and how I made it work. I want to focus in a little on that in this post.

If you are unfamiliar with XState I highly recommend checking it out. It has helped me develop better interfaces, components and part where it generates hundreds of tests for you given a few lines of code is just the icing on the cake.

The problem

When defining the configuration of a state machine that models the user's interaction with your application you also instruct your test model (created using xstate/test) what action to perform that simulates the event. Here's an example of test modeled for a form with a single text input and a button:

Application code:

<div>
  <div id="message">Guess the right value?!</div>
  <input type="text" value="" id="txtInput" />
  <button>Try!</button>
</div>
<script>
  document.querySelector('button').addEventListener('click', () => {
    const guessedRight = document.getElementById('txtInput').value === "777"
    document.getElementById('message').innerHTML = guessedRight
      ? "You guessed right!"
      : "Nope! Try again!"    
  })
</script>

State machine that models the test:

import { createMachine } from 'xstate'

const testMachine = createMachine({
  initial: 'fillOutForm'
  states: {
    fillOutForm: {
      on: { CLICK_BUTTON: 'guessed' },
      meta: {
        test: () => expect(document.getElementById('message').innerHTML)
                      .toBe("Guess the right value?!")
      }
    },
    guessed: {
      meta: {
        test: (_, e) => {
          const guessedRight = e.value === "777"
          expect(document.getElementById('message').innerHTML)
            .toBe(guessedRight ? "You guessed right!" : "Nope! Try again!")
      }
    }
  }
});

Test model:

import { createModel } from '@xstate/test'

const testModel = createModel(testMachine).withEvents({
  CLICK_BUTTON: {
    cases: ["000", "777"],
    exec: (_, event) => {
      document.getElementById('txtInput').value === event.value
      document.querySelector('button').click()
    } 
})

As you can see, the map of events given to the model reflects 2 user interactions. One for entering a value and one for clicking the button. This is likely the most common use case and also recommended for complete end-to-end tests where nothing is mocked, intercepted, or substituted.

However we often want to test end-to-somewhere-in-the-middle. Is that just called an integration test? The most common example of this is intercepting and mocking network requests. And even for complete end-to-end tests you will find yourself in situations where you need control over the order-of-execution. And that can be difficult to wrap your mind around.

A good example of this is the usage of setTimeout. If you are using setTimeout in your application then at some point an event will occur that is not triggered by the user. Other examples of this are promises that resolve, setInterval, registered callbacks for events such as the window resizing, etc. Those events can not be simulated in your models event map as intuitively as user events.

Aside: This is where I see newcomers to XState go off the deep-end. They create a state machine in their application that handles some kind of asynchronous behaviour using an invoked service that takes their machine say from idle to loading and a final state of failure or success. All excited they start digging into model-based testing and use the same machine or a very similar one in their test. The first mistake made, (I did), is keeping the invoked service on the test machine.

However, the machine used in your application is not the the machine used in your test. The machine used in your application is interpreted (service) and transitions from state to state responding to user- or internal machine events. The test machine is never interpreted but analyzed and "converted" to a set (plan) of of possible transitions (paths).

Now that we know that non-user events can be a problem, lets put this problem in our example:

Application code:

<div>
  <div id="message">Guess the right value?!</div>
  <input type="text" value="" id="txtInput" />
  <button>Try!</buttton>
</div>
<script>
  document.querySelector('button').addEventListener('click', () => {
    const guessedRight = document.getElementById('txtInput').value === "777"

    document.getElementById('message').innerHTML = guessedRight
      ? "You guessed right!"
      : "Nope! Try again!"    

    setTimeout(() => {
      document.getElementById('message').innerHTML = "Guess again?!"
    }, 3000)
  })
</script>

State machine that models the test:

import { createMachine } from 'xstate'

const testMachine = createMachine({
  initial: 'fillOutForm'
  states: {
    fillOutForm: {
      ...
    },
    guessed: {
      on: { SET_TIMEOUT_DONE: 'guessAgain' },
      meta: {
        ...
      }
    },
    guessAgain: {
      meta: {
        test: () => waitFor(() =>
          expect(document.getElementById('message')
           .innerHTML
          ).toBe("Guess again?!")
      }
    },
  }
});

Test model:

import { createModel } from '@xstate/test'

const testModel = createModel(testMachine).withEvents({
  SET_TIMEOUT_DONE: ?????
  CLICK_BUTTON: {
    ...
})

And there we have it. Our test model's event map has an event for which we don't know how to write a simulation. Continuing off the deep-end described in the aside above: This is where developers will:

• Return a rejected or resolved promise
• Call setTimeout

And then look sad as their tests fail and they can't figure out why because intuitively this is how you would expect to take care of the event. For now you can just assign a void function to that event.

const testModel = createModel(testMachine).withEvents({
  SET_TIMEOUT_DONE: () => {}
  ...

Regardless, your test will fail at this point because the test model, after executing the test for the "guessed" state will execute the void handler for the "SET_TIMEOUT_DONE" event and continue with the test for the "guessAgain" state before the setTimeout in our code resolves.

This is where utility functions provided by most testing libraries such as Puppeteer, Cypress.io, and @testing-library come in. They will block an asynchronous test and retry an assertion or expectation until it succeeds or times out:

import { waitFor } from '@testing-libary/dom'
    guessAgain: {
      meta: {
        test: async () => waitFor (
          () => expect(
            document.getElementById('message').innerHTML
            ).toBe("Guess again?!")
        )
      }
    },

This probably covers 90% or more of all cases. Problem solved. This post isn't necessary.

It becomes a problem when the test for the state that dispatches the non-user event has additional blocking statements and your application moves on to it's "success" or "failure" state while your test is still busy checking for the "loading" state. When intercepting requests and immediately resolving them with a mock this issue also pops up. It's race issue. The order of execution could be:

1. Test model executes function defined for "SUBMIT" event.
2. Application code calls onClick handler.
3. onClick handler calls fetch or setTimeout.
4. Test model executes function defined for submitting.meta.test.

Case 1: The test is blocked asserting something other than being in submitting state:

1. Test for submitting.meta.test is blocking
2. The callback provided for the fetch promise or setTimeout in the application code resolves and the appliction's UI updates reflecting a successful or failed outcome.
3. Test for submitting.meta.test continues and asserts if the application's UI reflects a "loading" state.
4. Test fails.

Case 2: fetch or setTimeout are mocked and resolve immediately:

1. The callback provided for the fetch promise or setTimeout in the application code resolves and the application's UI updates reflecting a successful or failed outcome. 2.. Test for submitting.meta.test continues and asserts if the application's UI reflects a a "loading" state.
2. Test fails.

The solution

What if we could:

1. Intercept where the application calls fetch or setTimeout.
2. Block execution of the application code.
3. Let the test do what it needs to.
4. Let the test unblock the application code whenever it wants.

We can! Whether you are using Puppeteer, Cypress.io, @testing-library or any other library for testing. As long as you have access to the environment the application is running in from your test.

As I type this I'm not 100% sure how you would intercept calls to setTimeout with Puppeteer but I'm pretty sure it would be possible.

We're going to continue with our simpler set-up that we started out with. To block setTimeout from resolving we are going to work with an array buffer of promises that allows.

const getFancyPromise = () => {
  let resolvePromise = () => throw "Unresolved promise"

  const promise = new Promise(resolve) => {
    resolvePromise = resolve
  }

  return Object.assign(promise, { resolvePromise })
}

I'll admit this is hacky but it allows me to resolve the promise outside of it's context:

const promise = getFancyPromise();
..
// much later
promise.resolvePromise();

Let's write our version of setTimeout

const makeSetTimeout = (buffer: Promise[]) => {
  const __setTimeout = window.setTimeout

  window.setTimeout = (cb: () => void, ms: number) => __setTimeout(async => {
    while (buffer.length > 0) {
      const promise = buffer[0]

      if (promise ) {
        await promise
        buffer.shift()
      }
    }

    __setTimeout(cb, ms)
  }, 0)
}

Given an array buffer we assign to window.setTimeout a version of it that delays the execution of the callback function until all promises in the buffer are resolved. Once a promise resolves it is removed from the buffer (mutative!!). Arrays in JavaScript are reference types. Anything pushing to the buffer is mutating the same array in memory as our function shifting from it.

We can now change our event mapping to push to this buffer before the button click simulation:

Test model:

import { createModel } from '@xstate/test'

const testModel = createModel(testMachine).withEvents({
  SET_TIMEOUT_DONE: () => {},
  CLICK_BUTTON: {
    cases: ["000", "777"],
    exec: (_, event) => {
      buffer.push(getFancyPromise())

      document.getElementById('txtInput').value === event.value
      document.querySelector('button').click()
    } 
})

And in the test for the "guessAgain" state we can resolve the promise in the buffer:

import { waitFor } from '@testing-libary/dom'

...
    guessed: {
      meta: {
        test: (_, e) => {
          const guessedRight = e.value === "777"
          expect(document.getElementById('message').innerHTML)
            .toBe(guessedRight ? "You guessed right!" : "Nope! Try again!")

          buffer.forEach(promise => promise.resolve())
      }
    }

So now the order of execution is:

1. Model simulates button click
2. Application code updates UI with message with "You guessed right!" or "Nope! Try again!"
3. Application code calls setTimeout which blocks at promise.
4. Model executes guessed.meta.test.
5. Test resolves promise in buffer.
6. Intercepted setTimeout continues as is.
7. Model executes guessAgain.meta.test and is blocked using waitFor.
8. setTimeout resolves.
9. Test passes.

This is a simple strategy that can be applied if you are running in edge cases when dealing with non-user events modeled in your test machine. I've been using it primarily to gain control on when network requests should be allowed to continue whether they are mocked or not.

Feel free to ask any questions or join me on https://spectrum.chat/statecharts and stop touching your face!

Chautelly.