I'm a big fan of lean Kubernetes clusters. This means that I'm quite hesitant to run there some resource-intensive workload that can't be easily scaled. So what to do with Prometheus that fits such description?
Well, there's a pretty new feature called remote write receiver that can help us to run Prometheus somewhere else and keep Kubernetes clusters small.
So let's just quickly go through the basic setup with single Prometheus server running in standalone EC2 instance.
This tutorial is very AWS and Terraform specific but it does not necessarily means that you can't use it anywhere else.
Prometheus installation: summary
Let's summarize what we need in order to get this running:
- Security group with ingress rule for 9090 port
- S3 bucket for configuration files
- IAM role
- IAM permissions
- Instance profile with role attached
- EBS volume
- some cloud-init that puts all the pieces together and start the Prometheus
- internal hostname
Security group
So the story here is that some agent running inside the cluster will be accessing the Prometheus server. Such cluster has some Security Group attached so we can use it as the reference in our rules - var.source_security_group_id
resource "aws_security_group" "this" {
name = "ec2_prometheus_${var.environment}"
vpc_id = var.vpc_id
}
resource "aws_security_group_rule" "egress_all" {
type = "egress"
to_port = 0
protocol = "-1"
cidr_blocks = [
"0.0.0.0/0",
]
from_port = 0
security_group_id = aws_security_group.this.id
}
resource "aws_security_group_rule" "ingress_prometheus" {
type = "ingress"
to_port = 9090
protocol = "tcp"
source_security_group_id = var.source_security_group_id
from_port = 9090
security_group_id = aws_security_group.this.id
}
Note the
0.0.0.0/0egress rule. This is kinda classic rule that basically allow all outgoing communication. In our case we'll need it to pull Prometheus artifacts from GitHub. This might differ based on your configuration and security requirements.
S3 bucket for the configuration
S3 bucket is used as the store for prometheus config, rules config and the alertmanager config. We can also use userdata but it's no so convenient since every change in configuration files would replace the EC2 instance.
Hence we render configuration files, put them into s3 bucket and Prometheus instance will be pulling them as needed.
So the following snippet create one S3 bucket with 3 objects - config, rules and alertmanager config.
locals {
prometheus_config = yamlencode({
global = {
scrape_interval = "1m",
},
rule_files = [
"/etc/prometheus/prometheus.rules.yaml",
],
alerting = {
alertmanagers : [
{
static_configs = [
{
targets = [
"localhost:9093",
]
}
],
}
],
},
})
}
resource "aws_s3_bucket" "config" {
bucket = "${var.project_name}-${var.environment}"
}
resource "aws_s3_object" "prometheus_config" {
bucket = aws_s3_bucket.config.id
key = "prometheus.yaml"
content = templatefile(
"${path.module}/prometheus.config.yaml",
{
environment = var.environment,
s3_bucket = aws_s3_bucket.config.id,
config = local.prometheus_config,
}
)
force_destroy = true
}
resource "aws_s3_object" "prometheus_rules" {
bucket = aws_s3_bucket.config.id
key = "prometheus.rules.yaml"
content = templatefile(
"${path.module}/prometheus.rules.config.yaml",
{
environment = var.environment,
s3_bucket = aws_s3_bucket.config.id,
config = var.config_prometheus_rules,
}
)
force_destroy = true
}
resource "aws_s3_object" "alertmanager_config" {
bucket = aws_s3_bucket.config.id
key = "alertmanager.yaml"
content = templatefile(
"${path.module}/alertmanager.config.yaml",
{
environment = var.environment,
s3_bucket = aws_s3_bucket.config.id,
config = var.config_alertmanager,
}
)
force_destroy = true
}
Now let's check all the templates referenced there. The first one is the prometheus config.
# prometheus configuration
# environment: ${environment}
# bucket: ${s3_bucket}
---
${config}%
It does not contain anything special and same applies for the other files. This is the file for the rules
# prometheus configuration
# environment: ${environment}
# bucket: ${s3_bucket}
---
${config}
and this is the alertmanager configuration:
# alertmanager configuration
# environment: ${environment}
# bucket: ${s3_bucket}
---
${config}
The reason why we use just config placeholder is simple: we'd like to compose all the configuration with HCL, not YAML. Use local.prometheus_config for the reference. That's the magic.
IAM role, policy and instance profile
Everyone's playing some role. This EC2 instance is not different. It needs to communicate with s3 bucket and SSM (for the remote management) so we take this into account.
resource "aws_iam_role" "this" {
name = "ec2_prometheus_${var.environment}"
assume_role_policy = jsonencode({
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "ec2.amazonaws.com"
}
}]
Version = "2012-10-17"
})
}
resource "aws_iam_policy" "this" {
name = "ec2_prometheus_${var.environment}"
policy = jsonencode({
Statement = [
# SSM Session manager stuff
{
Effect = "Allow"
Action = [
"ssm:DescribeAssociation",
"ssm:GetDeployablePatchSnapshotForInstance",
"ssm:GetDocument",
"ssm:DescribeDocument",
"ssm:GetManifest",
"ssm:ListAssociations",
"ssm:ListInstanceAssociations",
"ssm:PutInventory",
"ssm:PutComplianceItems",
"ssm:PutConfigurePackageResult",
"ssm:UpdateAssociationStatus",
"ssm:UpdateInstanceAssociationStatus",
"ssm:UpdateInstanceInformation"
]
Resource = "*"
},
{
Effect = "Allow"
Action = [
"ssmmessages:CreateControlChannel",
"ssmmessages:CreateDataChannel",
"ssmmessages:OpenControlChannel",
"ssmmessages:OpenDataChannel"
]
Resource = "*"
},
{
Effect = "Allow"
Action = [
"ec2messages:AcknowledgeMessage",
"ec2messages:DeleteMessage",
"ec2messages:FailMessage",
"ec2messages:GetEndpoint",
"ec2messages:GetMessages",
"ec2messages:SendReply"
]
Resource = "*"
},
// s3 for the configuration
{
Effect = "Allow"
Action = [
"s3:GetBucketLocation",
"s3:ListAllMyBuckets",
]
Resource = "*"
},
{
Effect = "Allow"
Action = [
"s3:List*",
]
Resource = [
aws_s3_bucket.config.arn,
]
},
{
Effect = "Allow"
Action = [
"s3:*",
]
Resource = [
"${aws_s3_bucket.config.arn}/*",
]
},
]
Version = "2012-10-17"
})
}
resource "aws_iam_role_policy_attachment" "this" {
policy_arn = aws_iam_policy.this.arn
role = aws_iam_role.this.name
}
resource "aws_iam_instance_profile" "this" {
name = "ec2_prometheus_${var.environment}"
role = aws_iam_role.this.name
}
EC2 instance
Now we can put all the pieces together and create the EC2 instance itself. In the following snippet you can also see the EBS volume and its attachment to the instance.
resource "aws_ebs_volume" "prometheus_0" {
availability_zone = data.aws_subnet.this.availability_zone
size = var.volume_size
type = "gp2"
}
resource "aws_instance" "prometheus_0" {
ami = var.ami_id
vpc_security_group_ids = [
aws_security_group.this.id,
]
subnet_id = var.subnet_id
iam_instance_profile = aws_iam_instance_profile.this.name
instance_type = var.instance_type
tags = {
Name = "${var.environment}_prometheus_0"
}
user_data_base64 = base64encode(
templatefile(
"${path.module}/cloud.init.yaml",
{
s3_bucket = aws_s3_bucket.config.id,
environment = var.environment,
}
)
)
}
resource "aws_volume_attachment" "prometheus_0" {
instance_id = aws_instance.prometheus_0.id
volume_id = aws_ebs_volume.prometheus_0.id
device_name = "/dev/sdf"
}
Please note the user_data_base64 property in the EC2 instance definition. This section refers to the YAML template with the following content:
#cloud-config
# environment: ${environment}
runcmd:
# install AWS CLI, neeeded for downloading of configuration files
- |
apt-get update && apt-get install unzip -y
curl -Lo awscli.zip https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip
unzip awscli.zip
./aws/install
rm awscli.zip
# install prometheus binary
- |
curl -Lo prometheus.tar.gz https://github.com/prometheus/prometheus/releases/download/v2.33.1/prometheus-2.33.1.linux-arm64.tar.gz
tar -xvf prometheus.tar.gz
cp ./prometheus-2.33.1.linux-arm64/prometheus /usr/local/bin/prometheus
rm -rf ./prometheus-2.33.1.linux-arm64
rm -rf prometheus.tar.gz
# install alertmanager binary
- |
curl -Lo alertmanager.tar.gz https://github.com/prometheus/alertmanager/releases/download/v0.23.0/alertmanager-0.23.0.linux-arm64.tar.gz
tar -xvf alertmanager.tar.gz
mv ./alertmanager-0.23.0.linux-arm64/alertmanager /usr/local/bin/alertmanager
rm -rf alertmanager-0.23.0.linux-arm64
rm alertmanager.tar.gz
# vait for EBS volume
- |
while [ ! -b $(readlink -f /dev/nvme1n1) ];
do
echo "waiting for device /dev/nvme1n1"
sleep 5
done
# format volume
blkid $(readlink -f /dev/nvme1n1) || mkfs -t ext4 $(readlink -f /dev/nvme1n1)
# create a mount
mkdir -p /data
if ! grep "/dev/nvme1n1" /etc/fstab;
then
echo "/dev/nvme1n1 /data ext4 defaults,discard 0 0" >> /etc/fstab
fi
# mount volume
mount /data
# enable and start systemd services
- |
systemctl daemon-reload
systemctl enable prepare-prometheus.service && systemctl start prepare-prometheus.service && sleep 10
systemctl enable prometheus.service && systemctl start prometheus.service
systemctl enable alertmanager.service && systemctl start alertmanager.service
write_files:
- path: /usr/local/bin/prepare-prometheus
permissions: '0744'
content: |
#!/bin/sh
mkdir -p /etc/prometheus
aws s3 cp s3://${s3_bucket}/prometheus.yaml /etc/prometheus/prometheus.yaml
aws s3 cp s3://${s3_bucket}/alertmanager.yaml /etc/prometheus/alertmanager.yaml
aws s3 cp s3://${s3_bucket}/prometheus.rules.yaml /etc/prometheus/prometheus.rules.yaml
curl -X POST http://localhost:9090/-/reload || true
- path: /etc/systemd/system/prepare-prometheus.service
content: |
[Unit]
Description=Prepare prometheus / alertmanager configuration
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/bin/prepare-prometheus
# please note data.mount in dependencies
- path: /etc/systemd/system/prometheus.service
content: |
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target data.mount prepare-prometheus.service
[Service]
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yaml \
--storage.tsdb.path /data/ \
--web.enable-lifecycle \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries \
--enable-feature=remote-write-receiver
[Install]
WantedBy=multi-user.target
- path: /etc/systemd/system/alertmanager.service
content: |
[Unit]
Description=Alert Manager
Wants=network-online.target
After=network-online.target data.mount prepare-prometheus.service
[Service]
Type=simple
ExecStart=/usr/local/bin/alertmanager \
--config.file /etc/prometheus/alertmanager.yaml \
--storage.path=/data/
[Install]
WantedBy=multi-user.target
When you start the instance with such user data, it will download all the tools required and create systemd units for prometheus and alertmanager.
Last but not least, there's also prepare-prometheus service. This oneshot (run for completion) downloads configuration files and puts them to the respective path - /etc/prometheus. And since prometheus and alertmanager services have this helper as a dependency - it's gonna happer before the prometheus starts.
Internal hostname
We'll also need to assign some hostname to the EC2 instance. The reason is obvious - we don't want to edit all the upcoming configuration files every time we replace the instance with the new one.
resource "aws_route53_zone" "prometheus" {
name = "prometheus.local"
vpc {
vpc_id = var.vpc_id
}
}
resource "aws_route53_record" "prometheus_0" {
name = "p01.prometheus.local"
type = "A"
records = [
aws_instance.prometheus_0.private_ip,
]
zone_id = aws_route53_zone.prometheus.zone_id
ttl = 60
}
Please note the vpc block in aws_route53_zone.prometheus resource. This means that prometheus.local names will be resolvable in the VPC. We're gonna use this functionality in the upcoming chapters.
Connecting to the EC2 instance
INSTANCE_ID=$(aws ec2 describe-instances \
--filters 'Name=tag:Name,Values=dev-ap-south-1_prometheus_0' 'Name=instance-state-name,Values=running' \
--output text --query 'Reservations[*].Instances[*].InstanceId')
Forwarding Prometheus to localhost
aws ssm start-session --target "${INSTANCE_ID}" --document-name AWS-StartPortForwardingSession --parameters '{"portNumber":["9090"],"localPortNumber":["9090"]}'
Wrap
In this part we've gone through the basic configuration of standalone prometheus instance. Such setup does not require any manual configuration and it will handle even AMI updates - all the persistent data are stored on EBS.
In the next chapter I'm gonna show you the other side - agents running in the Kubernetes cluster.
Top comments (0)