Getting Started with Prometheus Alerting

#tutorial #mysql #docker #monitoring

Continuing with articles about monitoring with Prometheus, this time alerts will be configured to know when a server or app stopped by using Alertmanager. Through this article, you will learn how to set up notifications for a MySQL server.

Running MySQL using Docker

First, create a network:

$ docker network create prom-network

Run the following command to get MySQL up and running in a container:

$ docker run -d -p 3306:3306 --name mysql -e MYSQL_ROOT_PASSWORD=root --network prom-network mysql:8.3.0

Running MySQL Server Exporter with Docker

From the documentation of Prometheus, there are a number of libraries and servers which help in exporting existing metrics from third-party systems as Prometheus metrics.

Some of these exporters are maintained as part of the official Prometheus GitHub organization, those are marked as official, others are externally contributed and maintained.

For MySQL, you can use the official MySQL Server Exporter that supports:

MySQL >= 5.6.
MariaDB >= 10.3

To install it you can follow the instructions in the GitHub repository and you can run it in a Docker container as well.

First, grant permissions to the exporter to access the database. Login to your MySQL server and run the following queries:

$ mysql -u root -p -h 127.0.0.1

CREATE USER 'exporter'@'%' IDENTIFIED BY 'password' WITH MAX_USER_CONNECTIONS 3;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'%';

Then, create the configuration file (config.my-cnf), adding login details as follows:

[client]
user = exporter
password = password
host = mysql

And finally, initialize the container:

$ docker run -p 9104:9104 --name exporter --network prom-network -v $(pwd)/config.my-cnf:/cfg/config.my-cnf prom/mysqld-exporter:main --config.my-cnf=/cfg/config.my-cnf

Running Alertmanager using Docker

First, create the configuration file (alertmanager.yml), this file will contain details about how to send notifications and will look as follows:

route:
  receiver: 'mail'
  repeat_interval: 4h
  group_by: [ alertname ]


receivers:
  - name: 'mail'
    email_configs:
      - smarthost: 'smtp.gmail.com:587'
        auth_username: '<your-email>'
        auth_password: "<your-password>"
        from: '<your-email>'
        to: '<some-email>'

Replace <your-email> and <your-password> with your login details. If you have 2FA activated, generate an app password. And replace <some-eamail> with the email where you want to get notifications.

The configuration file above specifies that a Gmail account will be used to send alerts. Check the Alerting section in the documentation of Prometheus for more information.

Then, run the container:

$ docker run --name alertmanager -d -p 9093:9093 --network prom-network -v ./alertmanager.yml:/etc/alertmanager/config.yml quay.io/prometheus/alertmanager --config.file=/etc/alertmanager/config.yml

Running Prometheus in a container

After configuring the exporter and Alertmanager, run Prometheus in a container, but first, create the configuration file (prometheus.yml). You can start with the basic configuration example from the documentation and add the configuration needed for the exporter as described in the GitHub repository.

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

rule_files:
  - "rules.yml"
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'mysql'

    params:
      auth_module: [client]

    scrape_interval: 5s

    static_configs:
      - targets: ['mysql:3306']

    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
          # The mysqld_exporter host:port
        replacement: mysql-exporter:9104

This block in the configuration file refers to the Alertmanager.

rule_files:
  - "rules.yml"
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

The rules.yml file contains the set of rules that tell Prometheus when to send a notification. The file will look as follows:

groups:
  - name: MysqldExporter
    rules:
      - alert: MysqlDown
        expr: mysql_up == 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: MySQL down (instance {{ $labels.instance }})
          description: "MySQL instance is down on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

It will create an alert whenever the MySQL server is down and once the alert is created, it will send the notification to the Gmail account previously configured.

Check the Awesome Prometheus alerts website for more examples on how to set the rules.

After both configuration files are created, just run the container.

docker run -p 9090:9090 --network prom-network --name prometheus -v ./prometheus.yml:/etc/prometheus/prometheus.yml -v ./rules.yml:/etc/prometheus/rules.yml prom/prometheus

Now Prometheus is up and running, with Alertmanager configured. You can try it by stopping the MySQL server, an alert will be created and an email similar to the following will be sent to the specified email account.