Queueing with BullJS

Introduction

A queue is a data structure that places elements in a sequence, similar to a stack. The difference is the methodology of accessing the data. It operates using the LIFO concept, meaning the last item to be added to the stack will be the first one to get accessed whereas the queue operates using FIFO meaning the first item inserted will be the first one out of the queue.

Please find the illustrations below:

There is a multitude of ways Queues are used in developing the most common being Sequential execution of tasks/jobs. It allows for developers to add jobs on one end and after processing leaving on the queue on another end. It usually operates on FIFO (First In First Out) methodology, which means the first job to arrive in the queue will get executed early before moving on to the next job arriving in the queue. After execution, the job is removed from the queue on the other end.

What queues are good for:

1. It allows breaking down large amounts of jobs into smaller more manageable jobs.
2. It Gives you the ability to add multiple processors for a particular job. Meaning we can attach multiple consumers that can pick up jobs from the queue and execute them
3. More complex workflows can easily be visualized and implemented with multiple queues ensuring consistency and atomicity.

BullJS

BullJS is one of the many node packages that can be used to implement queues in your application. Like any other Queueing library, it utilises the FIFO methodology.

npm install bull

Bull is extremely fast in execution and much of it owes to the fact that it uses Redis as the underlying technology that backs it up. This means data generated or inserted on the queue is stored in memory for faster reads and writes. Redis also supports pub/sub mechanism that aids in receiving callbacks for failed, completed or failed jobs.

Example implementation for Queues using BullJS

There are two parts to this implementation. A Producer, one that will produce the jobs to be added to the queue and a Consumer whose task is to execute the tasks added by the producer.

A Consumer should inform the Producer once the job has completed execution.

First define the redis configuration

const defaultRedisConfig = {
  redis: {
    host: redisHost,
    port: redisPort,
    db: redisDatabaseNumber,
  },
};

Consumer logic

// Redis server Configuration 
const <queue-name> = new Bull("<queue-name>", defaultRedisConfig);

setQueues([
  new BullAdapter(<queue-name>),
  ...
]);

// Consumer processing logic
<queue-name>.process(async (job) => {
   // Processing logic
});

Producer logic

const Bull = require('bull', { redis: defaultRedisConfig });
const <queue-name> = new Bull('tigo-queue');
<queue-name>.add(<Job Data>);

For similar blogs to this please checkout the ClickPesa Engineering Blog on:

Medium
Dev.to
Hashnode