- What is Kafka?
- Brew Installation Commands
- Directory Structure
- Brew Run Commands
- Alternative Commands
- Testing Installation
- Summary
Apache Kafka is a popular open-source stream-processing software platform developed by the Apache Software Foundation. Installing and configuring Kafka can sometimes be daunting, but the process is significantly simplified thanks to Homebrew (or brew), a package manager for macOS.
This blog will guide you through installing Kafka on macOS using Homebrew.
What is Kafka?
“Apache Kafka acts as a decoupling layer between source and target systems, allowing source systems to produce data into Kafka, and target systems to consume data from Kafka, making the architecture more scalable and manageable.” Apache Kafka
Brew Installation Commands
Before installing Kafka using Homebrew, you must ensure that Homebrew is installed on your Mac. Once Homebrew is installed, you can use the brew install
command to install both Kafka and Zookeeper, as Kafka has a dependency on Zookeeper which will be resolved by Homebrew automatically.
Command: Install Homebrew
$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
Command: Install Kafka
$ brew install kafka
This command will install both Kafka and Zookeeper, as Zookeeper is a dependency for Kafka.
Directory Structure
After the installation is complete, the Kafka and Zookeeper binaries, configurations, and logs will be located in different directories based on your Mac’s chip.
Macs with Intel Processor:
- Binaries and scripts will be in
/usr/local/bin
- Kafka configurations will be in
/usr/local/etc/kafka
- Zookeeper configuration will be in
/usr/local/etc/zookeeper
- The
log.dirs
config (Kafka data) will be set to/usr/local/var/lib/kafka-logs
Macs with Apple Silicon Processor:
- Binaries and scripts will be in
opt/homebrew/bin
- Kafka configurations will be in
opt/homebrew/etc/kafka
- Zookeeper configuration will be in
opt/homebrew/etc/zookeeper
- The
log.dirs
config (Kafka data) will be set toopt/homebrew/var/lib/kafka-logs
You can modify the Kafka and Zookeeper configuration files to customize your setup.
Brew Run Commands
Once you have installed Kafka and Zookeeper, you can start them using the brew services command. This command will start Kafka and Zookeeper as background services, so they will continue running even if you close your terminal.
Command: Start Zookeeper and Kafka
$ brew services start zookeeper
$ brew services start kafka
Command: Stop Zookeeper and Kafka
$ brew services stop kafka
$ brew services stop zookeeper
Command: Restart Zookeeper and Kafka
$ brew services restart kafka
$ brew services restart zookeeper
Alternative Commands
While Homebrew provides a convenient way to manage services, you can also start Zookeeper and Kafka manually using the provided scripts. This involves running the zookeeper-server-start.sh
and kafka-server-start.sh
scripts with the appropriate configuration files.
Command: Start Zookeeper
$ /usr/local/bin/zookeeper-server-start /usr/local/etc/zookeeper/zoo.cfg
Command: Start Kafka
$ /usr/local/bin/kafka-server-start /usr/local/etc/kafka/server.properties
Closing the terminal window or using the Ctrl-C
command will stop the Kafka and Zookeeper services. It is important to stop the services properly to avoid any data corruption or loss.
Testing Installation
Testing the Kafka installation involves starting the environment, creating a topic, producing events, and consuming events. After successfully performing these tasks, it is important to clean up the environment by deleting the created topic and stopping the Kafka and Zookeeper services to ensure that the system is left in a clean state.
Command: Start Zookeeper and Kafka
$ brew services start zookeeper
$ brew services start kafka
Creating a topic in Kafka can be done using the kafka-topics -- create
command. After creating the topic, it is important to validate its creation by describing the topic using the command kafka-topics --describe
, which will display the details of the created topic.
Command: Create a Topic
$ kafka-topics --create --topic quickstart-events --bootstrap-server localhost:9092
$ kafka-topics --describe --topic quickstart-events --bootstrap-server localhost:9092
Producing new events in Kafka can be done using the kafka-console-producer
command. This will send a Hello World
event to the Kafka topic you created.
Command: Produce Events
$ kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
> Hello World 1.
> Hello World 2.
You can stop the producer client with Ctrl-C
at any time.
Consuming the events in Kafka can be done using the kafka-console-consumer
command. This will retrieve the Hello World
event from the Kafka topic you created.
Command: Consume Events
$ kafka-console-consumer --topic quickstart-events --from-beginning --bootstrap-server localhost:9092
> Hello World 1.
> Hello World 2.
You can stop the consumer client with Ctrl-C
at any time.
Cleaning up the environment after testing the Kafka installation involves several steps. First, you need to delete the topic you created using the kafka-topics
command, then you need to shut down Kafka and Zookeeper using the brew services stop
commands, and finally, you need to delete any data of your local Kafka environment, including any events you have created along the way, by removing the Kafka logs and Zookeeper data directories that were specified in the configuration files.
Command: Environment Cleanup
$ kafka-topics --delete --topic quickstart-events --bootstrap-server localhost:9092
$ brew services stop kafka
$ brew services stop zookeeper
$ rm -rf /tmp/kafka-logs /tmp/zookeeper /tmp/kraft-combined-logs
Summary
Installing Kafka on macOS is straightforward using Homebrew. With just a few commands, you can have Kafka and Zookeeper up and running on your machine.
Happy streaming!
Top comments (0)