Setting Up Kafka on MacOS M1 using Homebrew

Introduction

This guide provides a step-by-step walkthrough of installing Apache Kafka on a MacOS M1 system. However, before we can install Kafka, we need to ensure Homebrew is installed on your system.

Homebrew Installation

Homebrew is a package manager for MacOS that simplifies the installation of software. To install Homebrew, open Terminal and run the following command:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

This will install Homebrew on your machine. For more information about Homebrew and how to use it, check out the official Homebrew documentation.

Kafka Installation

With Homebrew installed, we can now install Java and Kafka. Run the following commands in your Terminal:

brew install java
brew install kafka

These commands install both Java and Kafka on your system.

Running Kafka and Zookeeper

To start Kafka, we need to run Kafka and Zookeeper services separately. Open two Terminal windows for this purpose.

Start Zookeeper:

zookeeper-server-start /opt/homebrew/etc/kafka/zookeeper.properties

Start Kafka:

kafka-server-start /opt/homebrew/etc/kafka/server.properties

Make sure that both services are running successfully before proceeding.

Creating a Kafka Topic

Before interacting with Kafka using any producer or consumer API, you need to create a Kafka topic. A topic is essentially a category or feed name to which records get published. Here is how to create a topic named 'foobar':

kafka-topics --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic foobar

Note: We no longer use the --zookeeper flag because newer Kafka versions for MacOS do not require it.

Testing Kafka Producer and Consumer APIs

To validate your Kafka setup, let's test the Kafka producer and consumer APIs:

Open two new Terminal windows.

In the first terminal, initialize a producer console for the 'foobar' topic and send some test messages:

kafka-console-producer --broker-list localhost:9092 --topic foobar
> foo
> bar

In the second terminal, initialize a consumer console for the 'foobar' topic. This will listen to the bootstrap server on port 9092 and the 'foobar' topic:

kafka-console-consumer --bootstrap-server localhost:9092 --topic foobar --from-beginning
foo
bar

If you see these outputs, congratulations! Kafka is now set up and running smoothly on your MacOS M1 system.

Conclusion

Setting up Kafka on a MacOS M1 system is a straightforward process, especially with the help of Homebrew. With Kafka properly installed, you can now use it to manage real-time data pipelines and streams. Happy streaming!