"Here we go, another article on setting up some thing in Docker 😒", you might think. This time it's Kafka. And I wouldn't write about it if it was that trivial.
Thanks to wurstmeister, we have separate docker images for Kafka and Zookeeper. But setting this up my way without docker compose is a task in and of itself.
Zookeper
We'll need two volumes for this guy. You know how it goes, go to "Volumes" side-menu and smash the "Add volume" button. Name the first one zookeeper_data
, and the second one - zookeeper_conf
.
Next go to the "Containers" side menu, click the "Add container" button. Set:
- name: zookeeper
- image: wurstmeister/zookeeper:latest
- publish port: 2181:2181
- add volumes:
- /opt/zookeeper-3.4.13/data to zookeeper_data
- /opt/zookeeper-3.4.13/conf to zookeeper_conf
As always, a handy screenshot for you:
Now destroy that "Deploy the container" button and wait a little.
Kafka
This guy will need just one volume. Name it kafka_volume
. Now pay attention when creating a container. Set:
- name: kafka
- image: wurstmeister/kafka:latest
- publish port: 9092:9092
- add volumes:
- /kafka to kafka_data
- bind /var/run/docker.sock to /var/run/docker.sock
-
env:
- KAFKA_ADVERTISED_HOST_NAME: 172.17.0.1
- KAFKA_ZOOKEEPER_CONNECT: 172.17.0.1:2181
- ZK: 172.17.0.1:2181
- HOST: 172.17.0.1:9092
- KAFKA_ADVERTISED_PORT: 9092
ZK
and HOST
envs are just for shortening commands later on.
And deploy.
Communicate
There is already a guide on how to use this stuff, but there is a mistake in the consuming part.
Create topic
But let's start with creating a topic. Go to ">_ Console" of the Kafka container. Or execute command below in your terminal session
docker exec -it kafka /bin/bash
Once inside container create topic just as in above mentioned guide:
kafka-topics.sh --create --topic topic \
--partitions 4 --zookeeper $ZK --replication-factor 1
kafka-topics.sh --describe --topic topic --zookeeper $ZK
You'll see messages about topic creation and it's description:
Created topic topic.
Topic: topic PartitionCount: 4 ReplicationFactor: 1 Configs:
Topic: topic Partition: 0 Leader: 1002 Replicas: 1002 Isr: 1002
Topic: topic Partition: 1 Leader: 1002 Replicas: 1002 Isr: 1002
Topic: topic Partition: 2 Leader: 1002 Replicas: 1002 Isr: 1002
Topic: topic Partition: 3 Leader: 1002 Replicas: 1002 Isr: 1002
Consumer
The guide says to run this:
kafka-console-consumer.sh --topic=topic --zookeeper=$ZK
But if you try to do that you'll get an error about zookeeper is not a recognized option
.
So the correct command is this:
kafka-console-consumer.sh --topic=topic --bootstrap-server=$HOST
It didn't freeze, it just waits for messages to come.
Producer
Open another console or terminal session and connect to the kafka container. Execute:
kafka-console-producer.sh --topic=topic \
--broker-list=`broker-list.sh`
Now you can type some messages here and it'll show in another session where we have a consumer 🚀
Bonus
Inside kafka_app in the project repo, you'll find two short files with producer and consumer. This is just to show how we can use Kafka in Python:
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python consumer.py
In another terminal session:
source venv/bin/activate
python producer.py
Now look at the terminal with our python consumer and you'll see a topic, message key and value.
Also, if you didn't close previous sessions with the console consumer, you'll see the same message there. And publishing from the console producer will deliver a message to both console and python consumers.
Now you have one more tool in your toolbox! Go use this stuff 💪
Top comments (2)
nice walkthrough, the best tutorial at the moment on the internet about kafka docker portcontainer
thank you, I'm glad it was useful