DEV Community

Leandro Lima
Leandro Lima

Posted on

Using Apache Kafka with Node.js: A Tutorial on Building Event-Driven Applications

Using Apache Kafka with Node.js: A Tutorial on Building Event-Driven Applications

Apache Kafka is a powerful, open-source event streaming platform used to build distributed, real-time applications. When combined with Node.js, a lightweight JavaScript runtime, developers can build event-driven applications in a fraction of the time. In this tutorial, we'll explore how to use Apache Kafka and Node.js together, and how to create a simple event-driven application.

Prerequisites

Before we dive into using Apache Kafka and Node.js, you’ll need to make sure you have the following prerequisites installed on your machine:

  • Node.js version 12+
  • Node Package Manager version 6+
  • Apache Kafka 2.4+

You will also need a basic understanding of Node.js, JavaScript, and Apache Kafka. You can find more information about these topics in the following blog posts:

Setting up the Environment

Before we dive into the tutorial, you'll need to set up your environment. To begin, install Apache Kafka with the command:

$ sudo apt-get install kafka
Enter fullscreen mode Exit fullscreen mode

Next, start up the Kafka service with the commands:

$ sudo service kafka start
Enter fullscreen mode Exit fullscreen mode

Now that you have Kafka installed and running, let’s move on to installing the Node modules we’ll need for our application. You’ll need kafka-node and its dependencies. To install these, simply run the following command:

$ npm install kafka-node
Enter fullscreen mode Exit fullscreen mode

Writing the Code

Now that we have all of the prerequisites installed and our environment set up, it’s time to start writing our code. To begin, create a new file called producer.js, and add the following code:

const kafka = require('kafka-node');

// Create a new producer
const producer = new kafka.Producer();

// Connect to Kafka broker
producer.connect();

// Send a message to Kafka
producer.send(['my-topic'], [{message: 'Hello, Kafka!'}], (err, data) => {
  if (err) console.error(err); 
  else console.log(data);
});
Enter fullscreen mode Exit fullscreen mode

This code creates a new producer, connects to the Kafka broker, and sends a message to the my-topic topic.

Next, create a new file called consumer.js, and add the following code:

const kafka = require('kafka-node');

// Create a new consumer
const consumer = new kafka.Consumer();

// Connect to Kafka broker
consumer.connect();

// Listen to the 'my-topic' topic
consumer.on('message', (message) => {
  console.log(message);
});
Enter fullscreen mode Exit fullscreen mode

This code creates a new consumer, connects to the Kafka broker, and listens for messages on the my-topic topic.

Now that we’ve written the code for both the producer and the consumer, let’s move on to setting up an event-driven application.

Building the Event-Driven Application

To create an event-driven application, we will use Node.js to listen for events, and Apache Kafka to send and receive the events. We will combine the producer and consumer from the previous section to create a simple event-driven application.

First, create a new file called app.js and add the following code:

const kafka = require('kafka-node');

// Create producers
const producerA = new kafka.Producer();
const producerB = new kafka.Producer();

// Connect to Kafka broker
producerA.connect();
producerB.connect();

// Set up event listener
const messageListener = (message) => {
  if (message === 'A') producerA.send(['my-topic'], [{message: 'Hello, Kafka!'}], (err, data) => {
    if (err) console.error(err); 
    else console.log(data);
  });
  else if (message === 'B') producerB.send(['my-topic'], [{message: 'Hello, Kafka!'}], (err, data) => {
    if (err) console.error(err); 
    else console.log(data);
  });
};

// Create consumer
const consumer = new kafka.Consumer();

// Connect to Kafka broker
consumer.connect();

// Listen to the 'my-topic' topic
consumer.on('message', messageListener);
Enter fullscreen mode Exit fullscreen mode

This code creates two producers, connects them to the Kafka broker, and sets up an event listener. The event listener listens for events on the my-topic topic, and sends a message if one of the events is received.

Now that we’ve written the code for the application, let’s move on to running the application.

Running the Application

To run the application, you’ll need to start the Apache Kafka server. To do this, run the command:

$ sudo service kafka start
Enter fullscreen mode Exit fullscreen mode

Once the Apache Kafka server is running, you can start the application with the command:

$ node app.js
Enter fullscreen mode Exit fullscreen mode

Now that the application is running, we can send messages to the my-topic topic and see the results in the console. To do this, run the command:

$ kafka-console-producer --broker-list localhost:9092 --topic my-topic
Enter fullscreen mode Exit fullscreen mode

This command allows us to send messages directly to the my-topic topic. When you enter a message, you should see the results in the console.

Conclusion

In this tutorial, we’ve explored how to use Apache Kafka and Node.js together to create an event-driven application. We’ve seen how to set up the environment, write the code, and run the application. With Apache Kafka and Node.js, developers can quickly and easily create powerful distributed applications.

Top comments (0)