Remote writing your Prometheus metrics to Grafana Cloud

Setup Grafana Cloud

Register to Grafana Cloud.

Create API Key for Grafana Cloud

We will create an API Key to allow our local Prometheus to remote write to Grafana Cloud.

• Head over to your Grafana Cloud Portal and select Send Metrics on Prometheus.

• If you scroll above, you should see the section for API Key. Click on Generate now and create an API Key with the Role MetricsPublisher.

• Copy the Prometheus config and save it locally as values.yaml as we will use it to install our Prometheus Helm Chart. The url and username should be unique for every user.
  The password in both snippet should be filled with your API key.

Local Setup

I used Rancher Desktop but most of Kubernetes distribution should work similarly. We will deploy Prometheus using kube-prometheus-stack with Helm.

• Add Helm repository ```

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update

Before installing Prometheus, we will tweak the values.yaml we created earlier. We will not directly paste the username and password(API Key) to the file. Instead, we will create a secret and reference that from Prometheus Config.

$ kubectl create secret generic grafanacloudsecret \
--from-literal=username= \
--from-literal=password=''

Now, edit the values.yaml. Here is the edited version. Notice that the username and password is referencing the secret. Also I have added extra parameter which will attach extra labels to the metrics sent from this local cluster.

```yaml


prometheus:
  prometheusSpec:
    remoteWrite:
    - url: "https://prometheus-prod-10-prod-us-central-0.grafana.net/api/prom/push"
      basicAuth:
          username:
            name: grafanacloudsecret
            key: username
          password:
            name: grafanacloudsecret
            key: password
    replicaExternalLabelName: "__replica__"
    externalLabels: {cluster: "mylocalcluster"}

Now, let's install our Prometheus.

$ helm install prometheus prometheus-community/kube-prometheus-stack -n prometheus --create-namespace -f values.yaml

Check the metrics on Grafana Cloud

Go back to your Grafana Cloud and open the Grafana Cloud Billing/Usage. If you wait awhile, we should see some metrics being ingested in Grafana Cloud.

On the left pane, Go to Explore. In the Metrics browser, type in container_memory_usage_bytes. We can see the memory usage in my local Kubernetes cluster. Notice that there is an extra label named cluster: "mylocalcluster" which we added in our values.yaml. This can be used as an identifier for where this data is coming from.

Reference

• https://grafana.com/docs/grafana-cloud/kubernetes/prometheus/remote_write_helm_operator/
• https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack