In my previous article, we looked at how we can quickly set up Prometheus and Node Exporter.
But in that setup, Prometheus and Node Exporter processes are running in the foreground and it won't start the process after a reboot of the VM.
We are going to fix that issue in this article.
Prerequisites
Setup an EC2 instance of type t2.small
Ubuntu 22.04 LTS as AMI
30 GB of hard disk space
Open ports 22 for SSH, 9090 for Prometheus and 9100 for Node Exporter
Installation
Prometheus
Login to your EC2 instance
$ ssh -i prometheus.pem ubuntu@3.80.133.119
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.2 LTS
Release: 22.04
Codename: jammy
Create a new user for managing the Prometheus process
Upon executing the command it will create a user called prometheus without a home directory and shell as /bin/false
$ sudo useradd --shell /bin/false --no-create-home prometheus
Create a new directory for storing our Prometheus configuration file and update the ownership of the directory
$ sudo mkdir -p /etc/prometheus
$ sudo chown prometheus:prometheus /etc/prometheus
Create another directory for storing Prometheus time series data and update the ownership of the directory
$ sudo mkdir -p /var/lib/prometheus
$ sudo chown prometheus:prometheus /var/lib/prometheus
Download and extract the latest Prometheus release from their GitHub page
$ wget https://github.com/prometheus/prometheus/releases/download/v2.46.0/prometheus-2.46.0.linux-amd64.tar.gz
$ tar -xzvf prometheus-2.46.0.linux-amd64.tar.gz
Change the directory and install Prometheus and Promtool binaries
$ cd prometheus-2.46.0.linux-amd64
$ sudo install prometheus /usr/local/bin
$ sudo install promtool /usr/local/bin
Change the permission of these binaries to prometheus user
$ sudo chown prometheus:prometheus /usr/local/bin/prometheus
$ sudo chown prometheus:prometheus /usr/local/bin/promtool
Check the version of prometheus or promtool binaries
$ prometheus --version
prometheus, version 2.46.0 (branch: HEAD, revision: cbb69e51423565ec40f46e74f4ff2dbb3b7fb4f0)
build user: root@42454fc0f41e
build date: 20230725-12:31:24
go version: go1.20.6
platform: linux/amd64
tags: netgo,builtinassets,stringlabels
$ promtool --version
promtool, version 2.46.0 (branch: HEAD, revision: cbb69e51423565ec40f46e74f4ff2dbb3b7fb4f0)
build user: root@42454fc0f41e
build date: 20230725-12:31:24
go version: go1.20.6
platform: linux/amd64
tags: netgo,builtinassets,stringlabels
Copy the consoles and console_libraries directory which is used for dashboarding and visualization
$ sudo cp -rv consoles /etc/prometheus
$ sudo cp -rv console_libraries /etc/prometheus
Change the ownerships of the directories recursively to prometheus user
$ sudo chown -R prometheus:prometheus /etc/prometheus/consoles
$ sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
Copy the configuration file to the above-mentioned directory and change its ownership to prometheus user
$ sudo cp prometheus.yml /etc/prometheus
$ sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
Create a systemd unit file for the Prometheus service
$ sudo vi /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus \
--web.console.templates /etc/prometheus/consoles \
--web.console.libraries /etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target
Start the prometheus process and enable the process at boot
$ sudo systemctl daemon-reload
$ sudo systemctl start prometheus
$ sudo systemctl enable prometheus
Check the prometheus service status
$ sudo systemctl is-active prometheus
active
$ sudo systemctl is-enabled prometheus
enabled
Open http://3.80.133.119:9090 in the browser and you can see the Prometheus expression browser
Node Exporter
Create a new user for managing the node exporter process
$ sudo useradd --shell /bin/false --no-create-home node_exporter
Download and extract the latest Node Exporter release from their GitHub page
$ wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
$ tar -xzvf node_exporter-1.6.1.linux-amd64.tar.gz
Change the directory and install the node_exporter binary
$ cd node_exporter-1.6.1.linux-amd64
$ sudo install node_exporter /usr/local/bin
Change the ownership of the binary to node_exporter user
$ sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
Check the version of the node_exporter binary
$ node_exporter --version
node_exporter, version 1.6.1 (branch: HEAD, revision: 4a1b77600c1873a8233f3ffb55afcedbb63b8d84)
build user: root@586879db11e5
build date: 20230717-12:10:52
go version: go1.20.6
platform: linux/amd64
tags: netgo osusergo static_build
Create a systemd unit file for the Node Exporter service
$ sudo vi /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
Start the node_exporter process and enable the process at boot
$ sudo systemctl daemon-reload
$ sudo systemctl start node_exporter
$ sudo systemctl enable node_exporter
Check the status of the node_exporter process
$ sudo systemctl is-active node_exporter
active
$ sudo systemctl is-enabled node_exporter
enabled
Open http://3.80.133.119:9100/metrics in your browser to view all the metrics
Configuration
Prometheus
Now we have installed Prometheus and Node Exporter on the server and we need to modify the Prometheus configuration file to add new targets to scrape metrics
In the installation step, we already copied a default Prometheus configuration file to the /etc/prometheus location. You can edit this file or replace it with the below contents.
Here I have added a new job called "node_exporter" to scrape metrics from the same VM, instead of using localhost I have used the private IP of the instance
$ sudo vi /etc/prometheus/prometheus.yml
global:
scrape_interval: 15s
scrape_timeout: 10s
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "node_exporter"
static_configs:
- targets: ["172.31.93.117:9100"]
Validate the configuration file using the promtool
$ promtool check config /etc/prometheus/prometheus.yml
Checking /etc/prometheus/prometheus.yml
SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax
Restart the Prometheus service to effect new configuration changes
$ sudo systemctl restart prometheus
$ sudo systemctl status prometheus
Open the expression browser and navigate to Status -> Targets to view the newly added "node_exporter" target
We can view an example metric called "node_memory_Active_bytes" using the expression browser
Select the Graph option to view it as a graph
That's all for now
Reference
https://prometheus.io/docs/prometheus/latest/getting_started/
https://kodekloud.com/courses/prometheus-certified-associate-pca/
Top comments (0)