Table of Contents
- Download
- 1. Start zookeeper
- 2. Create a Kafka Broker
- 3. Create a Kafka topic
- 4. Create a Kafka Producer
- 5. Create a Kafka Consumer
- LOGS (DEBUG)
Download
curl -O http://packages.confluent.io/archive/5.4/confluent-community-5.4.0-2.12.zip
1. Start zookeeper
Navigate to the folder where confluent is installed. For me it was /home/divyaksh/Downloads/software/confluent-5.4.0
Linux
bin/zookeeper-server-start etc/kafka/zookeeper.properties
Windows
bin\windows\zookeeper-server-start.bat etc\kafka\zookeeper.properties
Error
Occurs if something is missing. I don't know what do do if this happens in linux.
Classpath is empty. Please build the project first using 'gradlew jarAll'
Solution
- Go to
bin/kafka-run-class - Add some code. Go to a line above this script (do not delete anything)
rem Classpath addition for core
add the below code
rem Classpath addition for LSB style path
if exist %BASE_DIR%\share\java\kafka\* (
call :concat %BASE_DIR%\share\java\kafka\*
)
2. Create a Kafka Broker
Linux
bin/kafka-server-start etc/kafka/server.properties
Windows
bin\windows\kafka-server-start.bat etc\kafka\server.properties
3. Create a Kafka topic
Linux
bin/kafka-topics --create --topic sampleTopic --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092
Windows
bin\windows\kafka-topics.bat --create --topic sampleTopic --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092
Warning
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both
Thus, create topics with names not haveing . or _.
--partition n:
- Number of partitions
- How much storage is needed (how big my message is)
- Nature of parallel processing
Here we have a simple workload of a single producer and a single consumer (n = 1)
--replication-factor f : number of copies of each partition in the cluster. You create multiple brokers and a in each you make a copy of the messages. Higher the replication factor, more number of copies. No sense of keeping f=10 when there is only 1 broker. in our example f = 1
--bootstrap-server: it takes 2 arguments ip-address and port-number. In our example we are running the zookeeper on local machine and the default port number of zookeeper is 9092.
Listing the topics
$ bin/kafka-topics --list --bootstrap-server localhost:9092
__confluent.support.metrics
sampleTopic
sample_topic
4. Create a Kafka Producer
Linux
bin/kafka-console-producer --topic sampleTopic --broker-list localhost:9092 < <path to some file>
Windows
bin\windows\kafka-console-producer.bat --topic sampleTopic --broker-list localhost:9092 < <path to some file>
--broker-list: same as --bootstrap-server.
NOTICE
$ bin/kafka-console-producer --help
This tool helps to read data from standard input and publish it to Kafka.
So you have to input the file using < operator before giving <path to some file>
5. Create a Kafka Consumer
Linux
bin/kafka-console-consumer --topic sampleTopic --bootstrap-server localhost:9092 -from-beginning
Windows
bin\windows\kafka-console-consumer.bat --topic sampleTopic --bootstrap-server localhost:9092 -from-beginning
OUTPUT
$ bin/kafka-console-consumer --topic sampleTopic --bootstrap-server localhost:9092 -from-beginning
To: divyaksh.shukla@gmail.com
From: ideapad.divyaksh@gmail.com
Subject: This is a test email from msmtp
MSMTP says hello
I had given a test email file as an input to the producer.
LOGS (DEBUG)
You can find the logs at /tmp/kafka-logs/ to debug kafka. Windows C:\tmp\kafka-logs\
Top comments (0)