• There are 4 components of Kafka:
o Producer
o Consumer
o Broker: A broker is nothing but the server and a node, which manages the topic.
o Zookeeper: Zookeeper maintains the state of brokers, in case of any broker going down or coming up, it notifies the zookeeper and zookeeper informs to the producer and consumer so that producer and consumer are connected to live broker.
• Topic is a classification for the message type let's say data type.
• The given topic is splitter into many partitions.
• And these partitions are replicated across more than 1 broker, if the number of brokers is more than 1.
• For the given partitions, there will be only 1 lead node and rest will be the slave node.
• Messages from the producer can go into any partitions and hence gets replicated.
• If the messages are assigned with the key, then it ensures the message with the same key in the particular partitions.
• In this way, it makes sure that messages are sequenced in the partitions.
• Kafka client (producer/consumer) to Kafka data transfer happens through https which makes sure that data is encrypted using SSL certificate.
• Use case:
o Messaging: A messaging platform built in publisher-subscriber model.
o Website tracking: to get the real time report of user activity in the website.
o Log aggregation: To aggregate the logs from many services.
o Fault tolerance: Kafka platform enables the back up of the events along with regular synch up.
o Stream Processing: events coming from multiple services can be processed in real
time and then can be transferred to other platform like storm for further processing.
• Eamples:
o Twitter uses it to take the post from the registered users to another registered
user to repost and read and to unregistered user for read.
o LinkedIn uses it for online consumption along with offline analytics while being
sorted in Hadoop.
o Netflix uses it for real time monitoring and event processing.
Top comments (0)