Flip your tests

Automated tests are great. They can help you run through hundreds of input combinations in a matter of seconds, a task that would be prohibitively burdensome to test by hand.

In my experience, a typical test suite looks like this:

describe('my test suite', () => {
  it('should work with basic test case', async () => {
    const user = await UserFactory.create({});
    expect(user.id).toBe(null);
    expect(user.name).toBe(null);
  });
  it('should work with a long name', async () => {
    const user = await UserFactory.create({
      firstName: 'Pippilotta',
      middleName: 'Delicatessa',
      lastName: 'Windowshade Mackrelmint Ephraimsdaughter Longstocking',
    });
    expect(user.id).toBe(null);
    expect(user.name).toBe('Pippilotta Delicatessa Windowshade Mackrelmint Ephraimsdaughter Longstocking');
  });
});

This design reflects the order in which an engineer has approached the problem. Often, the test cases directly correspond to edge cases that the engineer has considered. Each test follows this approximate format:

• Suite: all tests related to a particular subject or service
  • Test: condition A
    • Set up test case
    • Validate results
  • Test: condition B
    • Set up test case
    • Validate results
  • More tests to cover desired set of conditions.

However, there are a few drawbacks to this style:

• High cost of adding new tests. Each test setup must be copied into a new block in order to run the test.
• Lack of atomic visibility into code failures. Most modern test runners quit the test suite upon finding the first failure. If you're running several checks together as described above, you will only see the first issue.

Here's an alternate design:

describe('my test suite', () => {
  describe('basic test case', () => {
    let user;
    beforeAll(async () => {
      user = await UserFactory.create({});
    });
    it('should set null user id', async () => {
      expect(user.id).toBe(null);
    });
    it('should set null user name', async () => {
      expect(user.name).toBe(null);
    });
  });
  describe('with a long name', () => {
    let user;
    beforeAll(async () => {
      user = await UserFactory.create({
        firstName: 'Pippilotta',
        middleName: 'Delicatessa',
        lastName: 'Windowshade Mackrelmint Ephraimsdaughter Longstocking',
      });
    });
    it('should set null user id', async () => {
      expect(user.id).toBe(null);
    });
    it('should correctly form full name', async () => {
      expect(user.name).toBe(
        'Pippilotta Delicatessa Windowshade Mackrelmint Ephraimsdaughter Longstocking'
      );
    });
  });
});

• Suite: all tests related to a particular subject or service
  • Suite: condition A
    • beforeAll/Each: set up test case
    • Test: Validate result 1
    • Test: Validate result 2
  • Suite: condition B
    • beforeAll/Each: set up test case
    • Test: Validate result 1
    • Test: Validate result 2
  • More test suites to cover desired set of conditions.

This has several advantages:

• It's easier to debug when there are multiple failed tests. Sometimes you'll get one failure that triggers another. The previous approach, where you would only get a single failure message per test, would give you less information to help debug.
• All test failures are written in plain English. This makes it much easier to figure out what's going on.
• Passed tests are also written in plain English. This is also important! I believe very strongly in tracking the business decisions that led to a piece of code. Being forced to write out your tests in English makes it easier to realize when a piece of code is outdated and can be deleted.
• It's easier to perform nested test setups. If you want to test multiple levels of variation — say, check against multiple combinations of username, email address, and password — you can keep nesting your test suites as deep as you want to go, using beforeAll or beforeEach to add detail at each level. Just make sure you use afterAll or afterEach to clean up each case as you exit!
• It's easier to add placeholders for future tests. Many frameworks such as Jest have a modifier such as test.todo which allows you to write the title for a test without providing an implementation. This is much better than a // TODO comment, as your test runner will remind you that you still have some work left.

When you adopt a codebase, it's easy to fall into the patterns and conventions established by that codebase. With a little effort, though, you can start new habits that will reduce the amount of work you'll need to do in the future.