If you don't have commercial installation of Kafka that comes packed with monitoring solution, you have to build it on your own. You always need to have at least some tools to find what is going on in your service. Visualizing available metrics is indeed valuable option.

In this text, I will describe how to get these important metrics from Kafka and ship them to ElasticSearch where they can be visualized using Kibana. I will focus mainly on Kafka and Metricbeat configuration (how to get the metrics) rather than on visualization (make figures to your own taste).

Component list

• Kafka
• Jolokia
• Metricbeat
• ElasticSearch
• Kibana

Data pipeline

Kafka advertises runtime metrics using dedicated MBeans that can be made available using JMX interface. This is exactly the channel we will use here. One of the tools that is developed to collect metrics from various systems is Metricbeat that comes with prepared Kafka module that simply "knows what metrics Kafka provides". Behind the scenes, Metricbeat makes use of Jolokia that serves as bridge for JMX and provides metrics using HTTP/JSON.

Setting parts up

Kafka with Jolokia

Jolokia is a JMX-HTTP bridge giving an alternative to JSR-160 connectors. It is an agent based approach with support for many platforms. In addition to basic JMX operations it enhances JMX remoting with unique features like bulk requests and fine grained security policies.

In this article, I will not describe installation of Kafka. There is plenty of up-to-date documentation available. I am using rpm packaged Kafka distribution from Confluent.io. Let's focus on Jolokia.

We will be running Jolokia in agent mode. This is, we'll put Jolokia agent library on Kafka's class path, configure Jolokia in Kafka's configuration files and run it. This way, Jolokia will connect to Kafka monitoring MBeans and will provide metrics on HTTP interface.

• Download Jolokia JVM Agent - https://jolokia.org/download.html
• Save it into Kafka's lib folder /usr/share/java/kafka
• For easier configuration, either rename or make symlink /usr/share/java/kafka/jolokia-jvm-agent.jar

Now, we have the agent library in place and need to configure it

• Open /usr/bin/kafka-server-start and add KAFKA_JMX_OPTS

export KAFKA_JMX_OPTS="
-javaagent:/usr/share/java/kafka/jolokia-jvm-agent.jar=port=8778,host=localhost \
-Dcom.sun.management.jmxremote=true \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Djava.rmi.server.hostname=localhost \
-Dcom.sun.management.jmxremote.host=localhost \
-Dcom.sun.management.jmxremote.port=9999 \
-Dcom.sun.management.jmxremote.rmi.port=9999 \
-Djava.net.preferIPv4Stack=true"

• Start Kafka

At this point, you shall be able to see listening process on port 8778. You can try to get some numbers from it.

sudo netstat -ltnp
curl -s http://localhost:8778/jolokia/version | jq

Metricbeat

After installing Metricbeat, you find default configuration in /etc/metricbeat with Kafka module disabled. In Metricbeat, we need to do 2 things:

• Configure Kafka module for sampling, and
• Configure Metricbeat to send data to ElasticSearch and create Kibana artifacts.

To configure Kafka sampling

• Rename /etc/metricbeat/modules.d/kafka.yml.disabled to /etc/metricbeat/modules.d/kafka.yml to activate it.
• Open /etc/metricbeat/modules.d/kafka.yml. To read all available metrics, the configuration shall have this contents

# Module: kafka
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.6/metricbeat-module-kafka.html

# Kafka metrics collected using the Kafka protocol
- module: kafka
  metricsets:
    - partition
    - consumergroup
  period: 10s
  hosts: ["localhost:9092"]

  client_id: metricbeat

  # List of Topics to query metadata for. If empty, all topics will be queried.
  #topics: []

# Metrics collected from a Kafka broker using Jolokia
- module: kafka
  metricsets:
    - broker
  period: 10s
  hosts: ["localhost:8778"]

For detailed description of Metricbeat's Kafka module see documentation.

To configure proper forwarding of metrics to ElasticSearch

• Open /etc/metricbeat/metricbeat.yml, and configure elasticsearch output

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  index: "metricbeat-kafka-%{[agent.version]}-%{+yyyy-MM-dd}"

• Start Metricbeat service.

sudo systemctl start metricbeat

At this moment, Metricbeat shall start to sample Kafka metrics with 10 seconds interval and send them to ElasticSearch.

If you have Kibana installed, you can command Metricbeat to create visualizations and dashboard for you. For this, you need to configure Kibana endpoint parameters - where is your Kibana - and Kibana dashboard parameters - where in Kibana you want the artifacts to be created.

To create Kibana artifacts, run metricbeat setup command.

Version information

For this post, I used following component versions:

• Kafka - Confluent Community 5.4.1
• Jolokia 1.6.2
• Metricbeat 7.6.2
• ElasticSearch 7.6.0
• Kibana 7.6.0