DEV Community

Cover image for Installing Apache Kafka (Confluent edition)
Divyaksh
Divyaksh

Posted on

Installing Apache Kafka (Confluent edition)

#tutorial #machinelearning #beginners

Table of Contents

Download 

curl -O http://packages.confluent.io/archive/5.4/confluent-community-5.4.0-2.12.zip
Enter fullscreen mode Exit fullscreen mode

1. Start zookeeper

Navigate to the folder where confluent is installed. For me it was /home/divyaksh/Downloads/software/confluent-5.4.0

Linux

bin/zookeeper-server-start etc/kafka/zookeeper.properties
Enter fullscreen mode Exit fullscreen mode

Windows

bin\windows\zookeeper-server-start.bat etc\kafka\zookeeper.properties
Enter fullscreen mode Exit fullscreen mode

Error

Occurs if something is missing. I don't know what do do if this happens in linux.

Classpath is empty. Please build the project first using 'gradlew jarAll'
Enter fullscreen mode Exit fullscreen mode

Solution

  • Go to bin/kafka-run-class
  • Add some code. Go to a line above this script (do not delete anything) 
rem Classpath addition for core
Enter fullscreen mode Exit fullscreen mode

add the below code

rem Classpath addition for LSB style path
if exist %BASE_DIR%\share\java\kafka\* (
    call :concat %BASE_DIR%\share\java\kafka\*
)
Enter fullscreen mode Exit fullscreen mode

2. Create a Kafka Broker

Linux

bin/kafka-server-start etc/kafka/server.properties
Enter fullscreen mode Exit fullscreen mode

Windows

bin\windows\kafka-server-start.bat etc\kafka\server.properties
Enter fullscreen mode Exit fullscreen mode

3. Create a Kafka topic

Linux

bin/kafka-topics --create --topic sampleTopic --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Windows

bin\windows\kafka-topics.bat --create --topic sampleTopic --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Warning

WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both
Enter fullscreen mode Exit fullscreen mode

Thus, create topics with names not haveing . or _.

--partition n:

  • Number of partitions
    • How much storage is needed (how big my message is)
    • Nature of parallel processing

Here we have a simple workload of a single producer and a single consumer (n = 1)

--replication-factor f : number of copies of each partition in the cluster. You create multiple brokers and a in each you make a copy of the messages. Higher the replication factor, more number of copies. No sense of keeping f=10 when there is only 1 broker. in our example f = 1

--bootstrap-server: it takes 2 arguments ip-address and port-number. In our example we are running the zookeeper on local machine and the default port number of zookeeper is 9092.

Listing the topics 

$ bin/kafka-topics --list --bootstrap-server localhost:9092
__confluent.support.metrics
sampleTopic
sample_topic
Enter fullscreen mode Exit fullscreen mode

4. Create a Kafka Producer

Linux

bin/kafka-console-producer --topic sampleTopic --broker-list localhost:9092  < <path to some file>
Enter fullscreen mode Exit fullscreen mode

Windows

bin\windows\kafka-console-producer.bat --topic sampleTopic --broker-list localhost:9092 < <path to some file>
Enter fullscreen mode Exit fullscreen mode

--broker-list: same as --bootstrap-server.

NOTICE

$ bin/kafka-console-producer --help
This tool helps to read data from standard input and publish it to Kafka.
Enter fullscreen mode Exit fullscreen mode

So you have to input the file using < operator before giving <path to some file>

5. Create a Kafka Consumer

Linux

bin/kafka-console-consumer --topic sampleTopic --bootstrap-server localhost:9092 -from-beginning
Enter fullscreen mode Exit fullscreen mode

Windows

bin\windows\kafka-console-consumer.bat --topic sampleTopic --bootstrap-server localhost:9092 -from-beginning
Enter fullscreen mode Exit fullscreen mode

OUTPUT

$ bin/kafka-console-consumer --topic sampleTopic --bootstrap-server localhost:9092 -from-beginning
To: divyaksh.shukla@gmail.com
From: ideapad.divyaksh@gmail.com
Subject: This is a test email from msmtp

MSMTP says hello
Enter fullscreen mode Exit fullscreen mode

I had given a test email file as an input to the producer.

LOGS (DEBUG)

You can find the logs at /tmp/kafka-logs/ to debug kafka. Windows C:\tmp\kafka-logs\

Top comments (0)

Read next

msnmongare profile image

How to Set Git Username and Password in Git Bash?

Sospeter Mong'are -

mikeyoung44 profile image

AI Search Capabilities Hit Roadblock: Transformers Struggle with Large-Scale Path Finding

Mike Young -

dms_db_def6cfbeb08eaeda19 profile image

Errors as a learning

DMS DB -

itechburner profile image

CCSP Study Guide: How to Prepare for the CCSP Exam

Steve Smith -