Look ma, no Zookeeper

Background

Kafka always had dependency on Kafka since its inception for managing topics, managing ACLs, partitions, leader elections etc. While this is not a big issue you still need to maintain a quorum of Zookeeper (3 servers - recommended) for making sure Kafka runs smoothly in a clustered environment. That means maintaining a Zookeeper quorum which translates into running 3 instances of Zookeeper which adds to infrastructure cost as well as one more thing to maintain for DevOps.

Also Zookeeper had its share of limitations w.r.t. how much data you can store on each ZNode which is important if you are using ACLs on Kafka (details coming soon on this issue). Ever since KIP-500 was added to Kafka backlog, everyone was waiting for Kafka version which gets rid of Zookeeper. Kafka team had started moving in this direction from quite sometime now, with the new Kafka Admin API introduced in 2.1.0 version, it featured features like topic crating, adding/removing ACLs, deleting topics through Kafka without connecting to Zookeeper.

The day has now come with the introduction of Kafka 2.8 which gives us preview of Kafka without Zookeeper. *Please be aware that this feature is not intended to be used in production, Kafka team is still working on putting some missing pieces of the puzzle.

Getting started

Before getting started download Kafka binaries,

wget https://downloads.apache.org/kafka/2.8.0/kafka_2.13-2.8.0.tgz
tar -xzvf kafka_2.13-2.8.0.tgz
mv kafka_2.13-2.8.0 kafka
cd kafka

Step 1 - Generate cluster id

To get started for running Kafka without Zookeeper, a new cluster uuid needs to be generated.

./bin/kafka-storage.sh random-uuid

Step 2 - Format storage directories

The next step is to format storage directories, you will need to run this command on every broker needless to say you will run this command only on one broker if you are running in single mode. This step is new if you have worked with previous version of Kafka.

./bin/kafka-storage.sh format -t <uuid> -c ./config/kraft/server.properties

Replace with the id generated in Step bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic test-topic1.

Step 3 - Start Kafka Server

./bin/kafka-server-start.sh ./config/kraft/server.properties

There you go, if you have followed all the steps you should see Kafka running. Lets create a topic and send/receive messages to/from the topic.

Create Topic,

./bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic test-topic --partitions 1 --replication-factor 1

Run Kafka producer and send some messages,

./bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test-topic

Run Kafka consumer and verify if the messages are flowing through,

./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning

There you go! If all worked well, you should see messages on the test-topic.