How to write tests for applications that use MongoDB as a storage

MongoDB is one of the most popular databases right now and many people use it as primary storage on their applications in their pet projects or even in big production applications.

One of the main benefits of MongoDB is how flexible it is and how fast you can bring value. So I will try to show how to
write tests for your application to make sure the code you write will do what it is supposed to do.

Motivation

The motivation for writing this article is twofold.

The first reason for writing this article is to show that except the benefits of having your code tested ( confidence in the quality of your code, catch bugs before pushing code, etc ) it can also be quite as entertaining and educational as writing the actual code.

The second reason is for showing how we should write tests in isolation, meaning there should be no external interference that could probably skew test results.

I am going to show 3 different ways for testing:

• Mocking the MongoDB functionality
• Spinning a dedicated MongoDB instance with docker-compose for running the tests
• Use a mock MongoDB with Mongodb-In-Memory-Server

Technologies Used

During the article I am going to use:

• NodeJS
• Jest as a test runner
• Docker and docker-compose for setting locally Mongodb
• MongoDB In-Memory Server for mocking Mongodb

The code provided is written in Typescript but it's not a lot different from being in Javascript.

The code that needs testing

import { MongoClient, MongoClientOptions, Collection, ObjectId } from 'mongodb';

export function createClient(url: string, options?: MongoClientOptions) {
  return new MongoClient(url, options).connect();
}

export function createUserIndexes(client: MongoClient, database: string) {
  return Promise.all([
    client.db(database).createIndex('users', { email: 1 }, { unique: true }),
    client.db(database).createIndex('users', { occupation: 1 })
  ]);
}

interface UserDTO {
  _id: ObjectId;
  name: string;
  email: string;
  age: number;
  occupation: string;
  timestamp: string;
}

export class UserService {
  private collection: Collection;

  constructor(private client: MongoClient, database: string) {
    this.collection = this.client.db(database).collection('users');
  }

  createUser(user: Omit<UserDTO, 'timestamp' | '_id'>) {
    return this.collection.insertOne({
      ...user,
      timestamp: new Date().toISOString()
    });
  }

  getUser(email: string) {
    return this.collection.findOne<UserDTO>({ email });
  }

  getUsersByOccupation(occupation: string) {
    return this.collection.find<UserDTO>({ occupation }).toArray();
  }

  updateUser(
    email: string,
    payload: Partial<Omit<UserDTO, 'timestamp' | '_id'>>
  ) {
    return this.collection.updateOne({ email }, { $set: payload });
  }

  deleteUser(email: string) {
    return this.collection.deleteOne({ email });
  }
}

The above code consists of three components

• a function createClient that initializes and returns a MongoClient
• a function createUserIndexesthat creates indexes for the users collection
• and a class UserService that contains methods for interacting with users collection (create, delete, update user etc)

Method 1: Mocking MongoDB

The first method is about writing mocks that have the same interface as the actual MongoDB Driver. So in the tests, when the code calls .insertOne, a mock will emulate the functionality and spy on the arguments that the function was called with.

Let's see some examples.

The function createClient has as arguments the host url and the options that the MongoClient will be initialized with.

export function createClient(url: string, options?: MongoClientOptions) {
  return new MongoClient(url, options).connect();
}

Jest provides mocking functionality out of the box with jest.mock().

The setup for the tests:

jest.mock('mongodb');

describe('UserService', () => {
  const {
    constructorSpy,
    collectionSpy,
    createIndexSpy,
    databaseSpy,
    deleteOneSpy,
    findSpy,
    findOneSpy,
    insertOneSpy,
    updateOneSpy
  }: MongodbSpies = jest.requireMock('mongodb');

  beforeEach(() => {
    constructorSpy.mockClear();
    collectionSpy.mockClear();
    createIndexSpy.mockClear();
    databaseSpy.mockClear();
    deleteOneSpy.mockClear();
    findSpy.mockClear();
    findOneSpy.mockClear();
    insertOneSpy.mockClear();
    updateOneSpy.mockClear();
  });
  ...
});

Jest automatically will replace monogdb from the import { MongoClient } from 'mongodb' with the mock you provide in __mocks__/mongodb.ts. At jest.requireMock('mongodb'); I can get access to the spies that are specified in the mock and then use them inside our tests for asserting with what arguments the functions are getting called.

The mock file located at __mocks__/mongodb.ts:

export const constructorSpy = jest.fn();

export class MongoClient {
  constructor(url: string, options?: MongoClientOptions) {
    constructorSpy(url, options);
  }

  async connect() {
    return 'mock-client';
  }
}

The structure of the mock respects the interface of MongoDB Driver and exports a MongoClient with a connect method
that returns a string. It also includes a spy constructorSpy, with constructorSpy we can make sure that our constructor is called with the correct arguments.

Mock functions or known as 'spies' let you spy on the behavior of a function call.

An example of a test:

it('should connect and return a client', async () => {
  const url = 'mongodb://localhost:27017';
  const options = { keepAlive: true };
  const client = await createClient(url, options);

  expect(client).toBe('mock-client');
  expect(constructorSpy).toHaveBeenCalledWith(url, options);
});

With the same pattern, we can mock and verify all the methods are called with the correct arguments.

These methods are straightforward. We need to identify what methods from the MongoDB driver are being used by the
code and create mocks that emulate the functionality and assert that the code behaves as it should.

Pros of this method:

• This method gives us the ability to test the code that we have written in isolation of other factors like the MongoDB driver.
• Makes the tests more reliable as they don't rely on HTTP requests or connections with MongoDB in this case.
• Speed, once mocks are ready, it's fast to write and run the tests.

Cons of this method:

• Includes much more code, other than the actual implementation, creating the extra mocks. It's clear in the complete example that mocks are more lines of code than the actual UserService.
• Another problem with this method is that tests are relying a lot on the mocks. More times than I would like to admit my code misbehaves because of badly written or complex mocks.
• If a new breaking change is introduced to the MongoDB driver, you run the risk of not catching those breaking changes as your tests don't interact with the driver.
• Finally, tests can be a really good guide on how a function or a method is used and its signature.

In the below example, you can notice the createClient is returning a string. That's wrong and can be misleading to someone reading the tests.

  ...
  const client = await createClient(url, options);

  expect(client).toBe('mock-client');
  ...

Method 2: Using dedicated MongoDB with docker-compose

The second method in this article uses a different approach from the first one. Instead of mocking the MongoDB functionality, it's about creating a dedicated instance before running the tests, run the tests and then destroy it.

How we can do that? Create a MongoDB on demand keep it isolated and then just destroy it?

Here comes Docker and Docker Compose. I am not going to spend much time explaining Docker, but if you want I can write a separate blog about it.

The way for creating a MongoDB is through a manifest file called docker-compose.yaml

version: '3.9'

services:
  mongodb:
    image: mongo
    ports:
      - '27017:27017'
    volumes:
      - './seed.js:/docker-entrypoint-initdb.d/mongo-init.js:ro'

This docker-compose create a MongoDB service and attaches a seed file that runs on startup,
creates the needed collections and populates them with test data.

The commands for starting and stopping the MongoDB:

docker-compose up -d # -d (detach) is for running the service in the background

docker-compose down

Now the tests can run without mocks, they just need to point to the dedicated MongoDB.

The setup for the tests:

beforeAll(async () => {
  client = await createClient('mongodb://localhost:27017');
  userService = new UserService(client, database);
});

afterAll(async () => {
  await client.close();
});

beforeEach(async () => {
  await client.db(database).collection('users').deleteMany({
    name: 'test-user'
  });
});

BeforeAll tests create a client that connects to the docker-compose MongoDB.

AfterAll tests close the connection to MongoDB.

BeforeEach test deletes the test-user that was created during the tests, so each test is independent of previous data.

So all the tests are going to assert on real data.

Example:

it('should create needed indexes', async () => {
  const indexes = await createUserIndexes(client, database);

  expect(indexes).toEqual(['email_1', 'occupation_1']);
});

...

it('should return the correct user', async () => {
  const user = await userService.getUser('chef@email.com');

  expect(user).toEqual({
    _id: expect.any(ObjectId),
    name: 'mock-chef',
    email: 'chef@email.com',
    age: 27,
    occupation: 'chef',
    timestamp: '2021-09-29T15:48:13.209Z'
  });
});

Pros of this method:

• As you can see tests are much simpler and straightforward to write.
• Tests are more realistic and close to the actual use of our code. As noted before it's good to be able to read the tests and understand the code's behavior and functions/methods signatures.
• Finally, the integration between the UserService and the MongoDB driver is being tested, meaning if a breaking change is introduced, tests can catch it.

Cons of this method:

• Of course with this method, the process of running the tests, iterating on them, and setting up the environment is slower.
• It needs basic knowledge of Docker and Docker Compose for setting the testing environment and might get more difficult in more complex services. (I would highly recommend though investing some time on learning Docker and containers).

Method 3: Using In-Memory MongoDB server

The final method tries to combine both methods, 1 and 2. It uses an external package MongoDB In-Memory Server for our MongoDB.

As stated in the package description

This package spins up an actual/real MongoDB server programmatically from within NodeJS, for testing or mocking during development.

The tests in this method are quite similar to the tests from the Docker Method.

The setup for the tests:

beforeAll(async () => {
  mongod = await MongoMemoryServer.create();
  client = await createClient(mongod.getUri());
  await seedData(client, seed, database, 'users');
  userService = new UserService(client, database);
});

afterAll(async () => {
  await client.close();
  await mongod.stop();
});

beforeEach(async () => {
  await client.db(database).collection('users').deleteMany({
    name: 'test-user'
  });
});

The only difference is, that it needs to programmatically start the MongoDB Server and stop it at the end.

Pros of this method:

Some of the pros enlisted in both previous methods apply here

• Tests are much simpler and straightforward to write.
• Tests are more realistic and close to the actual use of our code.
• The integration between the UserService and the MongoDB driver is being tested.
• No complexity around setting up tests.
• Running and iterating tests is faster.

Cons of this method:

There are not many cons to this method.

I could just mention two things:

• First one is that there is not so much flexibility. An example of a missing feature for In-Memory Server is that there is no option for seeding data at the start, rather the tests need to do it programmatically.
• and secondly, this solution is specific to MongoDB, it might not be the case for the storage of your choice, having an In-Memory Server.

Conclusion

There are many ways to write your tests and make sure your code does what it is supposed to do, but like everything in software engineering, there is no such thing as one correct way. All the methods mentioned above have some benefits, but it all comes down to what each person or team values the most, or what you need to achieve by testing your code.

For example:

If you want to test your code in isolation and focus on the logic of your features then the 1st method would work for you.

If you want to test your code and how integrates and communicates with the system ( in this case with the MongoDB ) and get confidence that nothing breaks in between then second and third methods are better options for you.

My personal view is, go with what makes you feel more confident about your code. But either way please ALWAYS write tests, they are "life-saving".

You can find the complete example and code I shared in the article in Github 💻

Feel free to ask any questions/help in Github discussions or at the comments here ❓

If you liked or found the post useful just leave a ❤️