The article is about how to install Single Node multiple broker setup of Kafka and Zookeeper. This process is faster compared to installing kafka and zookeeper directly in Windows machine as both installation and configuration of kafka brokers and zookeeper are built-in using docker.
The article is useful for development and testing in windows machine.
Software's required
- Open Jdk-17 for windows https://www.oracle.com/in/java/technologies/javase/jdk11-archive-downloads.html
- Docker for windows https://docs.docker.com/desktop/install/windows-install/
- Kafka Client http://mirror.cogentco.com/pub/apache/kafka/3.7.0/kafka_2.12-3.7.0.tgz
Installation
1. Setup JDK 17 and configure PATH variables
- Skip this Step if JAVA is already installed.
- Open Windows Explorer, Right click on “This PC” and select Properties. Click on Advanced System Properties => Environment Variables.
- In System Variables section click on New Button and configure
JAVA_HOME = C:\jdk-17
-
Configure PATH variable. Select PATH and click on Edit. Add
%JAVA_HOME%/bin
-
Verify if the configuration is successfull. Open Command prompt and execute
java -version
C:\Users\suman>java -version
openjdk version "17" 2021-09-14
OpenJDK Runtime Environment (build 17+35-2724)
OpenJDK 64-Bit Server VM (build 17+35-2724, mixed mode, sharing)
2. Install Docker
- Download latest docker from https://docs.docker.com/desktop/install/windows-install/
- Follow normal installation steps to install docker desktop
3. Create docker-compose.yml file
- Download docker-compose.yml file from Git https://github.com/sumanthshastry/kafka-windows-docker
- Install kafka and zookeeper as per the configuration in docker file
docker-compose up -d — build
E:\kafka\kafka-windows-docker>docker-compose up -d --build
[+] Running 25/25
✔ zookeep3 Pulled 20.6s
✔ zookeep1 Pulled 20.6s
✔ kafka1 Pulled 63.1s
✔ zookeep2 8 layers [⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 20.6s
✔ 69692152171a Pull complete 11.2s
✔ ce2b89b60818 Pull complete 1.2s
✔ a3c211c6bdc9 Pull complete 0.9s
✔ 4c25606a3064 Pull complete 12.2s
✔ d7d77cede5c8 Pull complete 2.2s
✔ b2a5fc6a5ffe Pull complete 4.2s
✔ 49bd81bc3707 Pull complete 9.5s
✔ b6b7d3138b59 Pull complete 10.3s
✔ kafka3 Pulled 63.1s
✔ kafka2 11 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 63.1s
✔ 57168402cb72 Pull complete 27.7s
✔ 6edc125f768c Pull complete 48.4s
✔ 6086b74cc48f Pull complete 16.0s
✔ 04b3375cab83 Pull complete 17.2s
✔ f80eb2db9d48 Pull complete 23.1s
✔ 95d7897d98ce Pull complete 24.3s
✔ bd3de482b201 Pull complete 25.5s
✔ 0fbb79a570de Pull complete 26.8s
✔ 8592ceb925f7 Pull complete 27.9s
✔ be07a4b85081 Pull complete 41.1s
✔ 210860798c03 Pull complete 29.0s
[+] Running 6/7
- Network kafka-windows-docker_default Created 2.6s
✔ Container kafka-windows-docker-zookeep3-1 Started 1.4s
✔ Container kafka-windows-docker-zookeep2-1 Started 1.6s
✔ Container kafka-windows-docker-zookeep1-1 Started 1.7s
✔ Container kafka-windows-docker-kafka3-1 Started 1.5s
✔ Container kafka-windows-docker-kafka1-1 Started 1.6s
✔ Container kafka-windows-docker-kafka2-1 Started 1.8s
E:\kafka\kafka-windows-docker>docker-compose ps
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
kafka-windows-docker-kafka1-1 confluentinc/cp-kafka:7.5.0 "/etc/confluent/dock…" kafka1 34 seconds ago Up 31 seconds 0.0.0.0:9092->9092/tcp
kafka-windows-docker-kafka2-1 confluentinc/cp-kafka:7.5.0 "/etc/confluent/dock…" kafka2 34 seconds ago Up 31 seconds 9092/tcp, 0.0.0.0:9093->9093/tcp
kafka-windows-docker-kafka3-1 confluentinc/cp-kafka:7.5.0 "/etc/confluent/dock…" kafka3 34 seconds ago Up 32 seconds 9092/tcp, 0.0.0.0:9094->9094/tcp
kafka-windows-docker-zookeep1-1 zookeeper:3.4.14 "/docker-entrypoint.…" zookeep1 35 seconds ago Up 32 seconds 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp
kafka-windows-docker-zookeep2-1 zookeeper:3.4.14 "/docker-entrypoint.…" zookeep2 35 seconds ago Up 32 seconds 2181/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2182->2182/tcp
kafka-windows-docker-zookeep3-1 zookeeper:3.4.14 "/docker-entrypoint.…" zookeep3 35 seconds ago Up 32 seconds 2181/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2183->2183/tcp
E:\kafka\kafka-windows-docker>
4. Install Kafka Client
- Download Kafka Client from http://mirror.cogentco.com/pub/apache/kafka/3.7.0/kafka_2.12-3.7.0.tgz
- Unzip the file and navigate to bin/windows folder.
- Kafka Zookeeper compatibility matrix provides which versions are compatible
5. Create Topic
- Create new topic with topicName “test” with 3 partitions and 3 replication-factor
E:\kafka_cli\bin\windows>kafka-topics.bat --bootstrap-server localhost:9092 --topic test --create --partitions 3 --replication-factor 3
- Created topic test.
E:\kafka_cli\bin\windows>kafka-topics.bat --bootstrap-server localhost:9092 --topic test1 --create --partitions 3 --replication-factor 3
Created topic test1.
- List down the topics
E:\kafka_cli\bin\windows>kafka-topics.bat --bootstrap-server localhost:9092 --list
test
test1
- Describe a specific topic
E:\kafka_cli\bin\windows>kafka-topics.bat --bootstrap-server localhost:9092 --topic test --describe
Topic: test TopicId: NT5ydxyeTSaGFYmrH0xNaA PartitionCount: 3 ReplicationFactor: 3 Configs:
Topic: test Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
Topic: test Partition: 1 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
Topic: test Partition: 2 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
6. Produce and consume messages
- Produce few messages
E:\kafka_cli\bin\windows>kafka-console-producer.bat --broker-list localhost:9092 --topic test
>Hello
>from
>Bharath
>
- Consume messages from beginning
E:\kafka_cli\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
Hello
from
Bharath
Reference Links
http://kafka.apache.org/documentation.html
https://zookeeper.apache.org/doc/trunk/zookeeperStarted.html
https://www.conduktor.io/kafka/how-to-start-kafka-using-docker/
https://docs.confluent.io/platform/current/installation/versions-interoperability.html#zk
Top comments (0)