DEV Community

AutoMQ
AutoMQ

Posted on

Use Kafdrop to Manage AutoMQ

Preface
Kafdrop [1] is a simple, intuitive, and powerful web UI tool designed for Kafka. It allows developers and administrators to easily view and manage key metadata of Kafka clusters, including Topics, partitions, Consumer Groups, and their offsets. By providing a user-friendly interface, Kafdrop greatly simplifies the monitoring and management of Kafka clusters, enabling users to quickly obtain cluster status information without relying on complex command-line tools.

Thanks to AutoMQ's full compatibility with Kafka, it can seamlessly integrate with Kafdrop. By utilizing Kafdrop, AutoMQ users can also benefit from an intuitive user interface for real-time monitoring of Kafka cluster status, including Topics, partitions, Consumer Groups, and their offsets. This monitoring capability not only enhances problem diagnosis efficiency but also helps optimize cluster performance and resource utilization.
This tutorial will teach you how to start the Kafdrop service and integrate it with an AutoMQ cluster to monitor and manage the cluster state.

Image description

Prerequisites

  • Kafdrop Environment: AutoMQ cluster, JDK17, and Maven 3.6.3 or above.
  • Kafdrop can be run through JAR files, Docker deployment, or protobuf deployment. Refer to the official documentation [3].
  • Prepare 5 hosts to deploy the AutoMQ cluster. It is recommended to choose Linux amd64 hosts with 2 CPUs and 16GB of RAM and prepare two virtual storage volumes.
  • Download the latest official binary installation package from AutoMQ Github Releases to install AutoMQ. Below, I will first set up the AutoMQ cluster and then start Kafdrop. Install and start the AutoMQ cluster. Configure S3URL. Step 1: Generate the S3 URL. AutoMQ provides the automq-kafka-admin.sh tool to quickly start AutoMQ. Simply provide the S3 URL containing the required S3 access points and authentication information to start AutoMQ with one click, without the need to manually generate cluster IDs or perform storage formatting. 
### 命令行使用示例
bin/automq-kafka-admin.sh generate-s3-url \
--s3-access-key=xxx \
--s3-secret-key=yyy \
--s3-region=cn-northwest-1 \
--s3-endpoint=s3.cn-northwest-1.amazonaws.com.cn \
--s3-data-bucket=automq-data \
--s3-ops-bucket=automq-ops
Enter fullscreen mode Exit fullscreen mode

Note: You need to pre-configure an AWS S3 bucket. If you encounter errors, please ensure the parameters and format are correct.
Output Result
After executing this command, the process will automatically proceed through the following stages:

  1. Based on the provided accessKey and secret Key, test the core features of S3 to verify the compatibility between AutoMQ and S3.
  2. Generate the s3url based on identity information and access point information.
  3. Obtain the startup command for AutoMQ using the s3url. In the command, replace --controller-list and --broker-list with the actual CONTROLLER and BROKER that need to be deployed. Example results are as follows: 
############  Ping s3 ########################

[ OK ] Write s3 object
[ OK ] Read s3 object
[ OK ] Delete s3 object
[ OK ] Write s3 object
[ OK ] Upload s3 multipart object
[ OK ] Read s3 multipart object
[ OK ] Delete s3 object
############  String of s3url ################

Your s3url is:

s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=xxx&s3-secret-key=yyy&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA


############  Usage of s3url  ################
To start AutoMQ, generate the start commandline using s3url.
bin/automq-kafka-admin.sh generate-start-command \
--s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" \
--controller-list="192.168.0.1:9093;192.168.0.2:9093;192.168.0.3:9093"  \
--broker-list="192.168.0.4:9092;192.168.0.5:9092"

TIPS: Please replace the controller-list and broker-list with your actual IP addresses.
Enter fullscreen mode Exit fullscreen mode

Step 2: Generate a list of startup commands
Replace the --controller-list and --broker-list in the commands generated in the previous step with your host information. Specifically, substitute them with the IP addresses of the 3 CONTROLLERs and 2 BROKERs mentioned in the environment setup, and use the default ports 9092 and 9093.

bin/automq-kafka-admin.sh generate-start-command \
--s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" \
--controller-list="192.168.0.1:9093;192.168.0.2:9093;192.168.0.3:9093"  \
--broker-list="192.168.0.4:9092;192.168.0.5:9092"
Enter fullscreen mode Exit fullscreen mode

Output Result
Upon executing the command, a startup command for AutoMQ will be generated.

############  Start Commandline ##############
To start an AutoMQ Kafka server, please navigate to the directory where your AutoMQ tgz file is located and run the following command.

Before running the command, make sure that Java 17 is installed on your host. You can verify the Java version by executing 'java -version'.

bin/kafka-server-start.sh --s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" --override process.roles=broker,controller --override node.id=0 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override listeners=PLAINTEXT://192.168.0.1:9092,CONTROLLER://192.168.0.1:9093 --override advertised.listeners=PLAINTEXT://192.168.0.1:9092

bin/kafka-server-start.sh --s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" --override process.roles=broker,controller --override node.id=1 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override listeners=PLAINTEXT://192.168.0.2:9092,CONTROLLER://192.168.0.2:9093 --override advertised.listeners=PLAINTEXT://192.168.0.2:9092

bin/kafka-server-start.sh --s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" --override process.roles=broker,controller --override node.id=2 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override listeners=PLAINTEXT://192.168.0.3:9092,CONTROLLER://192.168.0.3:9093 --override advertised.listeners=PLAINTEXT://192.168.0.3:9092

bin/kafka-server-start.sh --s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" --override process.roles=broker --override node.id=3 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override listeners=PLAINTEXT://192.168.0.4:9092 --override advertised.listeners=PLAINTEXT://192.168.0.4:9092

bin/kafka-server-start.sh --s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" --override process.roles=broker --override node.id=4 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override listeners=PLAINTEXT://192.168.0.5:9092 --override advertised.listeners=PLAINTEXT://192.168.0.5:9092


TIPS: Start controllers first and then the brokers.
Enter fullscreen mode Exit fullscreen mode

Note: The node.id is automatically generated starting from 0 by default.
Step 3: Start AutoMQ
To start the cluster, sequentially execute the list of commands generated in the previous step on the pre-specified CONTROLLER or BROKER hosts. For instance, to start the first CONTROLLER process on 192.168.0.1, execute the first command template from the generated startup command list.

bin/kafka-server-start.sh --s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" --override process.roles=broker,controller --override node.id=0 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override listeners=PLAINTEXT://192.168.0.1:9092,CONTROLLER://192.168.0.1:9093 --override advertised.listeners=PLAINTEXT://192.168.0.1:9092
Enter fullscreen mode Exit fullscreen mode

Run in the background
If you need to run in background mode, please add the following code at the end of the command:

command > /dev/null 2>&1 &
Enter fullscreen mode Exit fullscreen mode

Start the Kafdrop service
In the previous process, we set up the AutoMQ cluster and obtained the addresses and ports of all broker nodes. Next, we will start the Kafdrop service.
Note: Ensure that the address where the Kafdrop service is located can access the AutoMQ cluster; otherwise, it will result in connection timeout issues.
In this example, I use the JAR package method to start the Kafdrop service. The steps are as follows:

  • Pull the Kafdrop repository source code: Kafdrop GitHub [4] 
git clone https://github.com/obsidiandynamics/kafdrop.git
Enter fullscreen mode Exit fullscreen mode
  • Use Maven to locally compile and package Kafdrop to generate the JAR file. Execute the following in the root directory: 
mvn clean compile package
Enter fullscreen mode Exit fullscreen mode
  • Start the service, specifying the addresses and ports of the AutoMQ cluster brokers: 
java --add-opens=java.base/sun.nio.ch=ALL-UNNAMED \
    -jar target/kafdrop-<version>.jar \
    --kafka.brokerConnect=<host:port,host:port>,...
Enter fullscreen mode Exit fullscreen mode
  1. Replace kafdrop-<version>.jar with the specific version, such as kafdrop-4.0.2-SNAPSHOT.jar.
  2. --kafka.brokerConnect=host:port,host:port requires you to specify the host and port for the specific broker nodes in the cluster. The console startup output is as follows:

Image description

If not specified, kafka.brokerConnect defaults to localhost:9092.

Note: Starting from Kafdrop 3.10.0, a ZooKeeper connection is no longer required. All necessary cluster information is retrieved through the Kafka management API.
Open your browser and navigate to http://localhost:9000. You can override the port by adding the following configuration:

--server.port=<port> --management.server.port=<port>
Enter fullscreen mode Exit fullscreen mode

Final effect

  1. Complete interface

Image description

Displays the number of partitions, number of topics, and other cluster state information.

  1. Creating a Topic

Image description

  1. Detailed Broker Node Information

Image description

  1. Detailed Topic Information

Image description

  1. Message Information Under the Topic

Image description

Summary
Through this tutorial, we explored the key features and functionalities of Kafdrop, as well as the methods to integrate it with AutoMQ clusters. We demonstrated how to easily monitor and manage AutoMQ clusters. The use of Kafdrop not only helps teams better understand and control their data flow but also enhances development and operational efficiency, ensuring a highly efficient and stable data processing workflow. We hope this tutorial provides you with valuable insights and assistance when using Kafdrop with AutoMQ clusters.
References
[1] Kafdrop：https://github.com/obsidiandynamics/kafdrop
[2] AutoMQ：https://www.automq.com/zh
[3] Kafdrop Deployment: https://github.com/obsidiandynamics/kafdrop/blob/master/README.md#getting-started
[4] Kafdrop project repository: https://github.com/obsidiandynamics/kafdrop

Top comments (0)

Read next

akarce profile image

ELK Stack Mastery: Building a Scalable Log Management System

Cey -

reene44444 profile image

[WebGL2.0][Basic]scene graph

Reene -

mairouche profile image

Using CDC or Outbox pattern in microservices for decentralized data propagation

Meidi Airouche -

bladerik profile image

Building a TypeScript Video Editor as a Solo Dev

Lukáš -