Hands On: Pull metrics into Kubernetes from anywhere and treat them generically with the Keptn Metrics Server

Metrics, Metrics, Everywhere

Metrics are everywhere because tool sprawl is real. If you work in anything beyond a startup, you'll most likely have multiple metric-storing backends. Multiple Prometheus instances, endless vendor solutions, homegrown databases etc.

What if...

What if you could just say "get me a metric called foo" and magically, it appeared?

That's precisely what the Keptn Metrics Server allows.

Install Prometheus

The first thing you'll need, of course, is at least one backend to store metrics. So install Prometheus now:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kube-prom-stack prometheus-community/prometheus

Install Keptn

helm repo add keptn https://charts.lifecycle.keptn.sh
helm repo update
helm install keptn keptn/keptn -n keptn-system --create-namespace --wait

Expose Prometheus and Get an Existing Metric

Now expose the Prometheus UI and get the name of a metric you would like to use.

Get the names of the services. Look for one called prometheus-server running on port 80 (that's the Prometheus user interface):

kubectl get svc

Port forward to access the UI on port 8080:

kubectl -n default port-forward svc/prometheus-server 8080:80

Go to http://localhost:8080 and in the search box, type prometheus_.

Choose a metric which has a non-zero value. For example:

prometheus_http_requests_total{code="200", handler="/-/ready", instance="localhost:9090", job="prometheus"}

Make a note of that PromQL. You'll need it later.

Create KeptnMetricsProvider

A KeptnMetricsProvider is how you define where metrics are coming from. One KeptnMetricsProvider is created for each "backend".

So create one now. I'm deciding to call it local-prometheus:

keptnmetricsprovider.yml

---
apiVersion: metrics.keptn.sh/v1alpha3
kind: KeptnMetricsProvider
metadata:
  name: local-prometheus
  namespace: default
spec:
  type: prometheus
  targetServer: "http://prometheus-server.default.svc.cluster.local:80"

Apply it:

kubectl apply -f keptnmetricsprovider.yml

See all KeptnMetricsProviders:

kubectl get keptnmetricsproviders

Create a KeptnMetric

A KeptnMetric is created to define what you want to retrieve, and from where (which leverages the KeptnMetricsProvider) and how often the metric should be retrieved (fetchIntervalSeconds).

The KeptnMetricsProvider and KeptnMetric must be in the same namespace.

keptnmetric.yml

---
apiVersion: metrics.keptn.sh/v1alpha3
kind: KeptnMetric
metadata:
  name: prometheus-http-requests-total
  namespace: default
spec:
  provider:
    name: local-prometheus
  query: 'prometheus_http_requests_total{code="200", handler="/-/ready", instance="localhost:9090", job="prometheus"}'
  fetchIntervalSeconds: 10

Show all Keptn Metrics with:

kubectl -n default get keptnmetrics

That's it. Use any metric generically regardless of backend

All of your metrics are now available on-cluster to use for whatever you need. Horizontal Pod Autoscaling, FinOps etc. etc.