In this articles, we will see how to implement apache kafka with node.js. Apache Kafka for Node.js Developers
Firstly, kafka is a streaming platform which implements the publish/subscriber model.
Further, If you want to Learn how kafka works under the hood.you can read this article series
Architecture of Kafka Series - Part 2
Core Concepts of Kafka Series - Part 3
Kafka and Node.js
On the other hand, you can think of why as a node.js developer, i need to learn about Kafka. there is a reason and need for that.
Mainly, we are in the development Era of Microservices. Most of the Companies prefer to build their applications using Microservices.
If you are a Node.js developer, you may also be in a situation where you need to build one. If you do some research on Microservices, you will find lot of resources referring Kafka along with Microservices.
Reason being, is Many of the UseCases for Microservices are Event Sourcing
Implementating Kafka for Event Sourcing
This is one of the popular usecases where one Microservice talk with another Microservice through Kafka.
Mainly, we will see how to implement the above usecase using Kafka and Node.js
- User Service - this service handles all the user business logics such as user creation, update and delete.
- Post Service - this service handles all the operations of posts for an user.
Event Sourcing comes when we an user is deleted, we need to delete all the posts related to user.
When Client make an API call, gateway sends the request to respective Microservice.
If the particular service has some data dependencies with other services, it will send the Data to Other services using Kafka(Event Sourcing).
Other Services SubScribes the Kafka Topics and get the data to do the business operations.
Demo
Complete Source Code can be found here
Summary
Apache Kafka can be used in lot of situations like Event Sourcing,Data Streaming and Message Queue. As a Backend developer, it is always good to know different possibilities to decide the best solution for solving the problem.
References :
Top comments (3)
I saw you used kafka-node any specific reason you used it over kafkaJs or node-rdkafka? We originally used kafka-node but switched to KafkaJs because it did support LZ4, while keeping the SSL configured the same as KafkaJs, unlike node-rdkafka which required files. Both node-kafka and node-rdkafka seem to have a problem maintaining.
This is fantastic stuff. It's neat post thank you for sharing. One suggestion I would give using I'd HighlevelProducer provides round robin way of posting messages to topic
Thanks
Kishore
Thank you for your feedback and suggestion . i will update it.