DEV Community

Randika Madhushan Perera
Randika Madhushan Perera

Posted on • Edited on

Prometheus Fundamentals (Lesson-01)

#tutorial #prometheus #devops

1. Getting Started

1.1 Introduction

We'll start with the basics, discussing below things,

  • What Prometheus is and how it functions
  • Installation and configuration.
  • Set up your own Prometheus server.
  • Examine the Prometheus data model, its storage methods, and the query language for interacting with data.
  • Visualization techniques for representing Prometheus metric data. Various data collection methods.
  • Alerting mechanisms.
  • We'll touch upon advanced topics like high availability, federation, and security considerations in Prometheus.

2. Prometheus Basics

2.1 What is Prometheus

Prometheus is an essential open-source tool for monitoring and alerting. Prometheus plays a critical role in gathering data about applications and systems, enabling visualization and real-time alerts based on this data. Understanding monitoring is crucial in managing a robust DevOps infrastructure; it allows you to quickly and easily comprehend what's happening within your systems.

Prometheus excels in the metric collection, storing crucial data about systems and applications on its server. This data can then be visualized through various means, such as dashboards with graphs and charts, providing actionable insights.

Moreover, Prometheus is adept at alerting, where it can react in real-time to changes in data. For instance, if a server goes down, Prometheus can issue an alert, notifying you to address the issue.

Use Cases

Metric collection: Collect Important metrics about your systems and applications in one place.

Visualization: Build dashboards that provide an overview of the health of your systems.

Alerting: Receive an email when something is broken.

As for its technical background, Prometheus is primarily written in Go, with some components in Java, Python, and Ruby. It's open-source under the Apache 2.0 license, meaning it's free for varied uses. Originally developed by Matt T. Proud and Julius Volz and sponsored by SoundCloud, it now thrives as a project maintained by various individuals and organizations.

For comprehensive information and full documentation on Prometheus, you can visit prometheus.io.

2.2 Prometheus Architecture

Prometheus is composed of various components, but the two most fundamental are the Prometheus server and exporters.

The Prometheus server acts as a central hub that gathers and stores metrics. Exporters, on the other hand, are agents that expose data about systems and applications for the server to collect.

Prometheus server: A central server that gathers metrics and makes them available.

Exporters: Agents that expose data about systems and applications for collection by the Prometheus server.

Prometheus Pull Model

A key aspect of Prometheus is its pull model for metric collection, this means the Prometheus server pulls metric data from exporters - agents do not push data to the Prometheus server.

Prometheus Architecture

Prometheus Architecture

  • Prometheus Server: Collects metrics data.
  • Exporters: Provide metric data for Prometheus to consume.
  • Client Libraries: Easily turn your custom application into an exporter that exposes metrics in a format Prometheus can consume.
  • Prometheus Pushgatewway: Allows pushing metrics to Prometheus for certain specific use cases.
  • Alertmanager: Sends alerts triggered by metric data.
  • Visualization Tools: Provide useful ways to view metric data. (Not necessary)

Inside the Prometheus server, there are multiple components, including storage for metric data. The server is the core of the entire Prometheus ecosystem. Alongside this, Prometheus employs client libraries for different programming languages, allowing the creation of custom exporters. For instance, a Java application can be turned into an exporter using the Prometheus Java client library, enabling the server to pull metrics from it.

Prometheus also includes a Pushgateway for scenarios where a pull model is not feasible, like short-lived batch jobs. This component acts as a middleman, allowing these jobs to push metrics to it, which Prometheus can then pull. Additionally, the Prometheus Alertmanager is responsible for sending out alerts triggered by metric data.

Visualization tools are another critical component of Prometheus, offering various ways to interact with and make decisions based on metric data. These tools can be part of Prometheus or external, like Grafana, providing charts and graphs for data analysis

3. Installation and Configuration

3.1 Intstallation

In this lesson, we guide you through the process of building a Prometheus server. There are several installation options for Prometheus, including using precompiled binaries from prometheus.io, building from source code on GitHub, or running it in a Docker container.

Step 01:

The first step involves creating a Prometheus user and group using the useradd command.

$ sudo useradd -M -r -s /bin/false prometheus
Enter fullscreen mode Exit fullscreen mode

Next, directories for storing configuration files and Prometheus data are created: /etc/prometheus and /var/lib/prometheus. Following this, the Prometheus archive is downloaded from prometheus.io and extracted.

$ sudo mkdir /etc/prometheus /var/lib/prometheus
Enter fullscreen mode Exit fullscreen mode

Step 02:

$ wget https://github.com/prometheus/prometheus/releases/download/v2.48.1/prometheus-2.48.1.linux-amd64.tar.gz
Enter fullscreen mode Exit fullscreen mode 
$ sudo tar -xzvf prometheus-2.48.1.linux-amd64.tar.gz
Enter fullscreen mode Exit fullscreen mode

The extracted binaries, prometheus and promtool, are then copied to /usr/local/bin/, and ownership of these binaries is set to the Prometheus user and group.

$ sudo cp prometheus-2.48.1.linux-amd64/{prometheus,promtool} /usr/local/bin/
Enter fullscreen mode Exit fullscreen mode

Set the permission to 'prometheus'

[root@ip ~]# ll /usr/local/bin/
total 238496
-rwxr-xr-x. 1 root root 125066649 Dec 29 17:40 prometheus
-rwxr-xr-x. 1 root root 119151581 Dec 29 17:40 promtool
Enter fullscreen mode Exit fullscreen mode 
$ sudo chown prometheus:prometheus /usr/local/bin/{prometheus,promtool}
Enter fullscreen mode Exit fullscreen mode

Now, copy the 'consoles' and 'console_libraries' files to /etc/prometheus

$ sudo cp prometheus-2.48.1.linux-amd64/{consoles,console_libraries} /etc/prometheus
Enter fullscreen mode Exit fullscreen mode

The next steps involve setting up the configuration file, prometheus.yml, and copying it to /etc/prometheus/

$ sudo cp prometheus-2.48.1.linux-amd64/prometheus.yml /etc/prometheus/
Enter fullscreen mode Exit fullscreen mode

Set the permission prometheus in /etc/prometheus dir and /var/lib/prometheus

$ sudo chown -R prometheus:prometheus /etc/prometheus

$ sudo chown -R prometheus:prometheus /var/lib/prometheus
Enter fullscreen mode Exit fullscreen mode

Step 03:

After ensuring proper file ownership, we test running Prometheus in the foreground to confirm it's ready to receive web requests. To make Prometheus more manageable, it's turned into a systemd service by creating a prometheus.service unit file, detailing how the service should run.

Test:

$ prometheus --config.file=/etc/prometheus/prometheus.yml
Enter fullscreen mode Exit fullscreen mode

If the test succeeds you can see "Server is ready to receive web requests." in the logs.

Create the unit file and the below content.

$ sudo vim /etc/systemd/system/prometheus.service
Enter fullscreen mode Exit fullscreen mode 
[Unit]
Description=Prometheus Time Series Collection and Processing Server
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
 --config.file /etc/prometheus/prometheus.yml \
 --storage.tsdb.path /var/lib/prometheus/ \
 --web.console.templates=/etc/prometheus/consoles \
 --web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Start and enable the Prometheus service:

$ sudo systemctl daemon-reload

$ sudo systemctl enable --now prometheus
Enter fullscreen mode Exit fullscreen mode

Make an HTTP request to Prometheus to verify it is able to respond:

$ curl localhost:9090
Enter fullscreen mode Exit fullscreen mode

You can also access Prometheus in a browser using the server's public IP address: http://PROMETHEUS_SERVER_PUBLIC_IP:9090

prometheus-front-end

3.2 Configuring an Exporter

We focus on configuring a Prometheus exporter, specifically the Node Exporter, which is an essential component in the Prometheus architecture. The Node Exporter is a type of Prometheus exporter designed to run on *NIX machines, collecting various system metrics. We'll be setting up a simple Linux server, installing the Node Exporter program, and configuring it as a systemd service. This setup allows the exporter to expose metrics about the machine, which the Prometheus server can then scrape and collect.

Once the Node Exporter is set up on our Linux server, the next crucial step is to configure the scrape_config in our Prometheus server's configuration file (prometheus.yml). The scrape_config section in the Prometheus configuration file specifies the list of targets for Prometheus to scrape. This is where we tell Prometheus where to find the exporters and how to connect to them to retrieve metric data.

For those wishing to follow along with the demonstration, you'll need to set up an additional Linux server.

Once this server is set up, the Node Exporter binary is downloaded, installed, and configured as a systemd service to ensure it's always running in the background.

Step 01:

Create a user called "node_exporter":

$ sudo useradd -M -r -s /bin/false node_exporter
Enter fullscreen mode Exit fullscreen mode

Download and extract the Node Exporter binary:

$ wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz

$ tar -xzvf node_exporter-1.7.0.linux-amd64.tar.gz
Enter fullscreen mode Exit fullscreen mode

Step 02:

Copy the Node Exporter binary to the appropriate location and set ownership:

$ sudo cp node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/
Enter fullscreen mode Exit fullscreen mode 
$ sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
Enter fullscreen mode Exit fullscreen mode

Step 03:

Create a systemd unit file for Node Exporter:

$ sudo vi /etc/systemd/system/node_exporter.service
Enter fullscreen mode Exit fullscreen mode 
[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Start and enable the node_exporter service:

$ sudo systemctl daemon-reload

$ sudo systemctl enable --now node_exporter
Enter fullscreen mode Exit fullscreen mode

You can retrieve the metrics served by Node Exporter like so:

$ curl localhost:9100/metrics
Enter fullscreen mode Exit fullscreen mode

After setting up the Node Exporter, we add a new job under scrape_configs in the prometheus.yml file, specifying the private IP address and port (9100 by default) of the new Linux server. This setup enables Prometheus to scrape metrics from the Node Exporter running on the Linux server.

Step 04: Configure Prometheus to Scrape Metrics

Log in to your Prometheus server and configure Prometheus to scrape metrics from the new server.

$ vim /etc/prometheus/prometheus.yml
Enter fullscreen mode Exit fullscreen mode

Locate the scrape_configs section and add a new entry under that section.

...
- job_name: 'Linux Server'
 static_configs:
 - targets: ['<PRIVATE_IP_ADDRESS_OF_NEW_SERVER>:9100']
...
Enter fullscreen mode Exit fullscreen mode

Reload the Prometheus config:

$ sudo killall -HUP prometheus
Enter fullscreen mode Exit fullscreen mode

Navigate to the Prometheus expression browser in your web browser using the public IP address of your Prometheus server: :9090.

Run some queries to retrieve metric data about your new server:

up

node_filesystem_avail_bytes
Enter fullscreen mode Exit fullscreen mode

In summary, this lesson covers setting up a Prometheus exporter, configuring the Prometheus server to scrape metrics from this exporter, and verifying that the metrics are successfully being scraped. This process is crucial for monitoring and gathering insights about system performance and health.

Top comments (0)

Read next

spwoodcock profile image

FastAPI, Pydantic, Psycopg3: the holy trinity for Python web APIs

Sam -

simplylizz profile image

Automating Cron Jobs in Docker with Ofelia: CVZilla's Experience

Anton Yanchenko -

dm8ry profile image

Load Testing PostgreSQL on Kubernetes: A YAML-Only Approach

Dmitry Romanoff -

xxzeroxx profile image

MySQL Terminal: Alter table, charset and delete

Antonio Silva -