Build Your Own JavaScript Test Framework (in TypeScript)

This is the first in a planned series of posts where I'll take some heavily used tools from the JavaScript ecosystem and attempt to build minimal versions from the ground up.

If you've worked with JavaScript for a while, you might be familiar with some of the more common test frameworks. The two that spring to mind for me are Mocha and Jest, but there are plenty of others out there with varying levels of popularity. 

These are powerful libraries, with great tooling built around them. If your aim is to effectively test an application you're working on, I strongly advise against building your own - just choose your favourite of the many existing options and get going. 

However if you're interested in how you could approach this challenge, keep reading!

What we'll build

We're going to use typescript to build a test framework called Pretzel Test 🥨. It'll be basic but functional, and will provide:

• A test API - ie: describe blocks, before/beforeEach blocks etc…
• An expectations API - ie: expect(value).toEqual(someExpectedValue)
• A test reporter to output results to the terminal
• CLI invocation

If you don't have time to follow along, the final code and an example using Pretzel Test are available on github.

Here's a snippet to demonstrate the API we're shooting for:

import { describe, expect } from "pretzel-test";
import { addNumbers } from "./util.js";

describe("addNumbers")
  .before(() => {
    // Before block
  })
  .after(() => {
    // After block
  })
  .beforeEach(() => {
    // Before each block
  })
  .afterEach(() => {
    // After each block
  })
  .it("should correctly add two positive numbers", () => {
    const expected = 10;
    const actual = addNumbers(2, 8);
    expect(actual).toEqual(expected)
  })
  .it("should correctly add two negative numbers", () => {
    const expected = -10;
    const actual = addNumbers(-2, -8);
    expect(actual).toEqual(expected)
  })

As you can see, Pretzel Test will use a chainable API rather than the common nested describe blocks style. This was an intentional decision; I wanted to explore alternatives to that common pattern, as I find nested describe blocks can become unwieldy and difficult to parse.

Part One: Project structure

We're going to build this in TypeScript. Let's get started. First up, create a new project in your preferred fashion. For example:

$ mkdir pretzel-test
$ cd pretzel-test
$ yarn init

Then we'll install a few dependencies:

$ yarn add typescript colors command-line-args glob lodash
$ yarn add --dev @types/colors @types/node @types/glob

In your project root, create the following directory structure:

📁pretzel-test
|- 📝tsconfig.json
|- 📁bin
   |- 📝cli.js
|- 📁src
   |- 📝describe.ts
   |- 📝expect.ts
   |- 📝main.ts
   |- 📝reporter.ts
   |- 📝runner.ts
   |- 📝types.ts

Open tsconfig.json and add the following:

{
  "compilerOptions": {
    "outDir": "./dist",
    "lib": ["es2015"]
  },
  "include": ["src"]
}

It's not a complex tsconfig.json, however it's worth taking a look at the typescript docs if you're unsure about what's going on there.

If you're coding along in TypeScript, keep in mind that you'll need to compile the code with tsc before running it from the dist folder.

Then, in src/main.ts we'll import and and export the user facing API of pretzel test. The functions we import don't exist yet, but we'll build them out.

// src/main.ts

import { describe } from "./describe";
import { expect } from "./expect";

export { describe, expect };

Part Two: src/types.ts

In types.ts we'll define the main types that are used throughout the project. Taking a read through this should help you understand how the test framework is structured. Enter the following in your types.ts file. I'll explain it further below.

// src/types.ts

import { test, before, beforeEach, after, afterEach } from "./describe";

export interface ChainableApi {
  currentTestGroup: TestGroup;
  it: typeof test;
  case: typeof test;
  before: typeof before;
  beforeEach: typeof beforeEach;
  after: typeof after;
  afterEach: typeof afterEach;
}

export interface TestGroup {
  description: string;
  tests: Test[];
  before: Function[];
  beforeEach: Function[];
  after: Function[];
  afterEach: Function[];
}

export interface Test {
  description: string;
  fn: Function;
  result: TestResult;
}

export interface Options {
  rootDir: string;
  matching: string;
}

export interface TestResult {
  passed: boolean;
  error: Error;
  beforeError: Error;
  afterError: Error;
  beforeEachError: Error;
  afterEachError: Error;
}

Let's look at the interfaces we've defined in this file:

ChainableApi

This interface represents the object that will be returned by a call to describe(). Further, any chained calls to before() beforeEach() after() afterEach() or it() will return an object implementing this same interface, which will allow users of the API to chain an arbitrary number of calls to the initial describe() function. 

The ChainableApi interface also references a currentTestGroup property, which we've declared as implementing the TestGroup interface.

TestGroup

Conceptually, a test group represents a parsed set of tests that begin with a call to describe() , and encompasses any chained methods on that call. 

When it comes time to run our tests, the descriptions and callbacks passed in to the describe API will be pulled out into an object implementing the TestGroup interface. 

To accomodate this, we've defined a description property of type string, to contain the test description passed to the initial describe() call. We've then defined four properties - before , beforeEach , after & afterEach - which each accept an array of functions. These properties will be used to reference the callbacks functions passed to their respective methods in the ChainableApi.

Finally, we define a tests property, which accepts an array of objects implementing the Test interface. 

Test

The Test interface is quite similar to TestGroup , but will store references for a single test as defined by a call to it(). it() will accept two arguments - a description, and a callback function that runs the test expectations. As such, we have another description property of type string & an fn property of type Function.

We also have a result property, of type TestResult which will be used to store the results of the individual test after it's been run.

TestResult

The TestResult interface contains a passed property that accepts a boolean, which will indicate if the test passed or failed.

The remainder of the fields on TestResult are used to keep track of any errors thrown when running the test.

Part Three: src/describe.ts

In this file we define the test API of Pretzel Test. This, combined with the expectations API are what (hypothetical) users of our framework would use to author their tests. Here's the code:

// src/describe.ts

import { ChainableApi, TestGroup } from "./types";
import { testRunner } from "./runner";

const getInitialTestGroup = (description: string): TestGroup => {
  return {
    description,
    tests: [],
    before: [],
    beforeEach: [],
    after: [],
    afterEach: []
  };
};

function before(fn: Function): ChainableApi {
  this.currentTestGroup.before.push(fn);
  return this;
}

function after(fn: Function): ChainableApi {
  this.currentTestGroup.after.push(fn);
  return this;
}

function beforeEach(fn: Function): ChainableApi {
  this.currentTestGroup.beforeEach.push(fn);
  return this;
}

function afterEach(fn: Function): ChainableApi {
  this.currentTestGroup.afterEach.push(fn);
  return this;
}

function test(description: string, fn: Function): ChainableApi {
  this.currentTestGroup.tests.push({
    description,
    fn,
    result: {
      type: null,
      error: null,
      beforeError: null,
      beforeEachError: null,
      afterError: null,
      afterEachError: null
    }
  });
  return this;
}

function describe(description: string): ChainableApi {
  const currentTestGroup: TestGroup = getInitialTestGroup(description);
  testRunner.pushTestGroup(currentTestGroup);
  return {
    currentTestGroup,
    it: test,
    case: test,
    before,
    beforeEach,
    after,
    afterEach
  };
}

export { describe, test, before, beforeEach, after, afterEach };

I'll run through the above function by function:

describe() 

The entry point to the API is the describe function, which accepts a description string as its single argument. First, the function builds a currentTestGroup object (Initially the currentTestGroup object will only store the description that has been passed to describe, with all other properties set to empty arrays). 

Next up, we call testRunner.pushTestGroup and pass in the current test group object. testRunner is an instance of the TestRunner class, which we haven't yet defined, however its job will be to collect and run each TestGroup - so we pass it a reference to the test group that has been created as a result of the describe call.

Finally, the describe function returns an object that implements the ChainableApi interface. It contains references to the chainable methods (before, beforeEach, after, afterEach & it) along with the current test group via the currentTestGroup property.

before(), beforeEach(), after & afterEach ()

These functions all behave in the same way. First, they push the callback that's passed as an argument into their respective property on the currentTestGroup object, and then return this.

Because these methods will always be chained to a describe() call, the this keyword in each method will refer to the parent object that the methods were called on (in this case, the object returned from the initial describe block). 

As such, these methods have access to the currentTestGroup object via this.currentTestGroup. By returning this at the end of each function, we allow an arbitrary number of these methods can be chained, and each will still be able to access currentTestGroup in the same way.

it()

The it method is pretty similar to the other chainable methods in behaviour, with a couple of notable differences. 

Firstly, it accepts a description argument along with a callback function. Second, rather than only pushing a callback function, it builds and pushes an object implementing the the full Test interface to the currentTestGroup.

Part Four: src/expect.ts

This is file is where we create our expectation API. For now, we'll keep this very simple, and only implement matchers for .toEqual() and .notToEqual(), however this could be extended to provide more functionality. Take a look:

// src/expect.ts

import "colors";
import * as _ from "lodash";

const getToEqual = (value: any) => {
  return (expectedValue: any) => {
    if (!_.isEqual(value, expectedValue)) {
      throw new Error(`Expected ${expectedValue} to equal ${value}`.yellow);
    }
  };
};

const getNotToEqual = (value: any) => {
  return (expectedValue: any) => {
    if (_.isEqual(value, expectedValue)) {
      throw new Error(`Expected ${expectedValue} not to equal ${value}`.yellow);
    }
  };
};

export function expect(value: any) {
  return {
    toEqual: getToEqual(value),
    notToEqual: getNotToEqual(value)
  };
}

The expect() function accepts a value of any type, returning an object with our toEqual() and notToEqual() expectation functions. If the expectations fail, they throw a error (which is caught and recorded by the testRunner.

We're cheating a little here and using Lodash's isEqual() method to perform the actual equality comparison, as it provides a deep equality check that's a bit tricky to code manually.

Part Five: src/runner.ts

The TestRunner class has a few responsibilities:

1. It serves as the entry point to Pretzel Test. When we later implement the cli script to start the test run, it will do so with a call to testRunner.run().
2. It initiates an instance of the Reporter class (which will be responsible for logging the test results to the console.
3. It locates and imports test files matching the glob pattern passed through as options.
4. It collects the test groups from the imported files, then loops over them and invokes the actual test functions, recording the results.

Here's the code:

// src/runner.ts

import * as glob from "glob";
import { Reporter } from "./reporter";
import { TestGroup, Test, Options } from "./types";
import { EventEmitter } from "events";

export class TestRunner extends EventEmitter {
  static events = {
    testRunStarted: "TEST_RUN_STARTED",
    testRunCompleted: "TEST_RUN_COMPLETED",
    afterBlockError: "AFTER_BLOCK_ERROR",
    testGroupStarted: "TEST_GROUP_STARTED",
    testGroupCompleted: "TEST_GROUP_COMPLETED",
    singleTestCompleted: "SINGLE_TEST_COMPLETED"
  };

  suite: TestGroup[];

  constructor(Reporter) {
    super();
    new Reporter(this);
    this.suite = [];
  }

  pushTestGroup(testGroup: TestGroup) {
    this.suite.push(testGroup);
  }

  buildSuite(options: Options) {
    const testFilePaths = glob.sync(options.matching, {
      root: options.rootDir,
      absolute: true
    });
    testFilePaths.forEach(require);
  }

  async runBeforeEachBlocks(test: Test, testGroup: TestGroup) {
    try {
      for (const fn of testGroup.beforeEach) await fn();
    } catch (error) {
      test.result.beforeEachError = error;
    }
  }

  async runTestFn(test: Test) {
    try {
      await test.fn();
      test.result.passed = true;
    } catch (error) {
      test.result.passed = false;
      test.result.error = error;
    }
  }

  async runAfterEachBlocks(test: Test, testGroup: TestGroup) {
    try {
      for (const fn of testGroup.afterEach) await fn();
    } catch (error) {
      test.result.afterEachError = error;
    }
  }

  async runTests(testGroup: TestGroup) {
    for (const test of testGroup.tests) {
      await this.runBeforeEachBlocks(test, testGroup);
      await this.runTestFn(test);
      await this.runAfterEachBlocks(test, testGroup);
      this.emit(TestRunner.events.singleTestCompleted, test);
    }
  }

  async runBefore(testGroup: TestGroup) {
    try {
      for (const fn of testGroup.before) await fn();
    } catch (error) {
      testGroup.tests.forEach(test => {
        test.result.beforeError = error;
      });
    }
  }

  async runAfter(testGroup: TestGroup) {
    try {
      for (const fn of testGroup.after) await fn();
    } catch (error) {
      this.emit(TestRunner.events.afterBlockError, error);
      testGroup.tests.forEach(test => {
        test.result.beforeError = error;
      });
    }
  }

  async runTestGroup(testGroup: TestGroup) {
    this.emit(TestRunner.events.testGroupStarted, testGroup);
    await this.runBefore(testGroup);
    await this.runTests(testGroup);
    await this.runAfter(testGroup);
    this.emit(TestRunner.events.testGroupCompleted, testGroup);
  }

  async run(options: Options) {
    this.buildSuite(options);
    this.emit(TestRunner.events.testRunStarted);
    for (const testGroup of this.suite) await this.runTestGroup(testGroup);
    this.emit(TestRunner.events.testRunCompleted);
  }
}

export const testRunner = new TestRunner(Reporter);

I won't go through this file function by function (or you'll be here all day), however there is one thing I'd like to point out. You'll see that the TestRunner class extends Nodes inbuilt EventEmitter. This gives us access to emit() and on(), which you'll see being used above, and in reporter.ts. This is how the testRunner communicates with the reporter, and triggers the reporter to log output to the console.

Part Five: src/reporter.ts

As you've hopefully seen above, the Reporter class is imported and initialised by the testRunner, with the testRunner passing itself as an argument to the Reporter constructor. The Reporter constructor then initialises a set of event listeners on the testRunner (using the EventEmitter .on() function, which in turn trigger callbacks that console.log the various testRunner events (passing test, failing test, etc).

We're also using the colors npm package to make the console output a little more interesting.

Here's the code:

// src/reporter.ts

import "colors";
import { TestRunner } from "./runner";
import { Test, TestGroup } from "./types";

const indent: string = "  ";

export class Reporter {
  testRunner: TestRunner;

  constructor(testRunner) {
    this.testRunner = testRunner;
    this.initEventListeners();
  }

  printSummary = () => {
    let totalCount: number = 0;
    let passedCount: number = 0;
    let failedCount: number = 0;
    this.testRunner.suite.forEach(testGroup => {
      totalCount += testGroup.tests.length;
      testGroup.tests.forEach(test => {
        if (test.result.passed) passedCount += 1;
        else {
          console.log(`\n ○ ${testGroup.description}. ${test.description}`.red);
          console.log(`\n${test.result.error.stack}`);
          failedCount += 1;
        }
      });
    });
    console.log(`\n Total tests run: ${totalCount}`.yellow);
    console.log(` Passing tests: ${passedCount}`.green);
    console.log(` Failing tests: ${failedCount}\n`.red);
  };

  handleTestGroupStarted = (testGroup: TestGroup) => {
    console.log(`\n ${testGroup.description}`.grey);
  };

  handleTestGroupCompleted = () => {};

  handleTestRunStarted = () => {
    console.log("\n [Pretzel 🥨]: Starting test run...".yellow);
  };

  handleTestRunCompleted = () => {
    console.log("\n [Pretzel 🥨]: Test run completed.\n".yellow);
    console.log("\n Summary:".yellow);
    this.printSummary();
  };

  handleAfterBlockError = error => {
    console.log("There was an error in an after block...");
  };

  handleSingleTestCompleted = (test: Test) => {
    if (test.result.passed) {
      console.log(`   ○ ${test.description} ✓`.grey);
    } else {
      console.log(`   ○ ${test.description} ✖`.red);
    }
  };

  initEventListeners() {
    const {
      testRunStarted,
      testRunCompleted,
      afterBlockError,
      singleTestCompleted,
      testGroupStarted,
      testGroupCompleted
    } = TestRunner.events;
    this.testRunner.on(testRunStarted, this.handleTestRunStarted);
    this.testRunner.on(testRunCompleted, this.handleTestRunCompleted);
    this.testRunner.on(afterBlockError, this.handleAfterBlockError);
    this.testRunner.on(testGroupStarted, this.handleTestGroupStarted);
    this.testRunner.on(testGroupCompleted, this.handleTestGroupCompleted);
    this.testRunner.on(singleTestCompleted, this.handleSingleTestCompleted);
  }
}

Part Seven: bin/cli.js

The final piece of the pretzel! This script, when combined with a little config in package.json, will allow our test framework to be invoked from the command line by users who have installed our package.

In this script, we use the command-line-args npm package to collect some required configuration options from the user:

• --rootDir sets the root tests directory
• --matching accepts a glob pattern to match test files (ie. **/*.test.js

If these options aren't provided, then we through an error.

Then, we import the testRunner (importantly, this is being imported from dist/, not src/), and initiate the test run by calling testRunner.run(options) with the provided user options.

#!/usr/bin/env node

const commandLineArgs = require("command-line-args");

const optionDefinitions = [
  { name: "rootDir", alias: "r", type: String },
  { name: "matching", alias: "m", type: String }
];

const options = commandLineArgs(optionDefinitions);

if (!options.rootDir) throw new Error("rootDir is a required argument");
if (!options.matching) throw new Error("matching is a required argument");

const { testRunner } = require("../dist/runner.js");

testRunner.run(options);

To allow this script to be invoked from the command line, we need to register it. We also need to point the package.json to the entry point (dist/main.js)Add the following to package.json:

{
  "name": "pretzel-test",
  "main": "dist/main.js",
  ...
  "bin": {
    "pretzel-test": "./bin/cli.js"
  }
}

And thats it! We're done!

Almost...

In order to use this package to run some tests on your machine, you'll need to npm link it (since pretzel-test isn't actually available on npm). From your pretzel-test project root run npm link.

Then from the root of the package you want to run pretzel-test in, run npm link pretzel-test. You should now be able to require('pretzel-test') as normal from within this package.

Alternatively, you can clone the Pretzel Test Example Repo that I've created, which provides an example of using the framework and how to set it up. Take a look at the readme for further instructions.

If you found this post useful, you can follow me on dev.to or twitter. I've also got a couple of side projects that you might like to check out:

• ippy.io - An app for creating beautiful resumes
• many.tools - A collection of useful utilities for designers and devs