In the last blog we walked through the process of setting up a Prometheus server with Grafana dashboard. In this blog we will be setting up the alertmanager in the prometheus server.
Structure of an alertmanager and the prometheus server
Defining rules for prometheus server
- In the last blog we created a
alerts.rules
config files, this file will contain all the rules for our prometheus server
cd /opt/prometheus/
vim alerts.rules
groups:
- name: Linux-disk-space
rules:
- alert: Linux-DiskSpaceAlert
expr: (node_filesystem_avail_bytes/node_filesystem_size_bytes)*100 < 20
for: 5m
annotations:
summary: "Disk space is running low"
description: 'The available disk space on {{ $labels.instance }} is less than 80percent'
- name: Linux-memory-utilisation
rules:
- alert: Linux-MemoryUsage
expr: ((node_memory_MemAvailable_bytes/node_memory_MemTotal_bytes)*100 ) < 20
for: 2m
annotations:
Summary: "High RAM Usage"
description: 'Memory usage on {{ $labels.instance }} is greater than 80percent'
- name: Linux-up
rules:
- alert: Linux-up-status
expr: up{job='node-exporter'} < 1
for: 2m
annotations:
Summary: "Linux machine/exporter down"
description: '{{ $labels.instance }} metrics could not be fetched machine may be down'
- name: Windows-memory-utilisation
rules:
- alert: Windows-MemoryUsage
expr: ((windows_os_physical_memory_free_bytes/windows_cs_physical_memory_bytes)*100 ) < 20
for: 2m
annotations:
Summary: "High RAM Usage"
description: 'RAM usage on {{ $labels.instance }} is higher than 80 percent'
- name: Windows-diskspace-utilisation
rules:
- alert: Windows-diskspace-utilisation
expr: (windows_logical_disk_free_bytes/windows_logical_disk_size_bytes)*100 < 20
for: 2m
annotations:
Summary: "Disk Space running low"
description: 'Diskspace utilisation on {{ $labels.instance }} is higher than 80 percent'
- name: Windows-up
rules:
- alert: Windows-up-status
expr: up{job='win-exporter'} < 1
for: 2m
annotations:
Summary: "Windows machine/exporter down"
description: '{{ $labels.instance }} metrics could not be fetched machine may be down'
Setting the alertmanager
- for this walkthrough we will be setting up gmail alerts using gmail smtp server
1. setting up gmail smtp server
To set up a gmail smtp server for prometheus we need a gmail account say sender@gmail.com
on which 2-step verification is enabled
- your app password is your
secret-key
which will be used inalertmanager.yml
asauth_password
2. config file for alertmanager
cd /opt/prometheus/
touch alertmanager.yml
vim alertmanager.yml
global:
resolve_timeout: 1m
route:
receiver: 'gmail-alert-notification'
group_wait: 10s
group_interval: 1m
repeat_interval: 1m
# sender@gmail is the gmail account on which you have generated your app password
receivers:
- name: 'gmail-alert-notification'
email_configs:
- to: 'receiver@gmail.com'
from: 'sender@gmail.com'
smarthost: smtp.gmail.com:587
auth_username: 'sender@gmail.com'
auth_identity: 'sender@gmail.com'
auth_password: 'secret-key'
3. installing the alertmanager docker image
We will be installing the docker image of alertmanager on the same machine on which we installed the prometheus server
docker run -d -p 9093:9093 -v /opt/prometheus/alertmanager.yml:/etc/alertmanager/alertmanager.yml --restart unless-stopped --name alertmanager prom/alertmanager
- verify that alertmanager is running by visiting
alertmanager-ip:9093
in your browser
Adding rules and alertmanager in the prometheus
cd /opt/prometheus/
vim prometheus.yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'node-exporter'
static_configs:
- targets: ['ubuntu_id:9100']
- job_name: 'win-exporter'
static_configs:
- targets: ['windows_ip:9182']
rule_files:
- "/etc/prometheus/alerts.rules"
alerting:
alertmanagers:
- static_configs:
- targets:
- "ubuntu_ip:9093"
# ubuntu_ip is the IP on which the alertmanager container is installed, use localcost:9093 if same as prometheus server
# if prometheus docker is already running run the following command
docker restart prometheus
# if not run the following
docker run -d -p 9090:9090 -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml -v /opt/prometheus/alerts.rules:/etc/prometheus/alerts.rules --restart unless-stopped --name prometheus prom/prometheus
Your prometheus and alertmanager is ready and can be verified by visiting prometheus_ip:9090
and alertmanager_ip:9093
on your browser
I hope this blog helped you understand how to setup prometheus with alertmanager. Stay connected for more such blogs
Top comments (0)