Add deployment annotations to grafana dashboards

When noticing spikes in exceptions on our dashboards, one of the first things we do is check if there has been a recent deployment. Checking to see if a deployment caused the spikes.

This can be straightforward, but if you have many teams deploying many apps all throughout the day, it may be a little trickier tracking down a related deployment.

I am going to talk you through how to add a grafana annotation to visualize when a deployment has taken place. This will be a simple annotation with some tags.

Here is what we are gonna create

First of all, I am going to be running this using my grafana playground which you can find here

Setup Grafana, Jaeger & Zipkin locally

Wesley Skeen ・ Nov 21 '22

#grafana #dotnet

Once you have that set up, there are a few steps we need to do to enable annotations to display on a dashboard.

Create a service account

Browse here and click Add service account. If you have the grafana playground running, you should be able to access the link.

Give it a name and the Editor role

Create a service account token

Click Add service account token

Then click Generate token

Once you have this token, copy it and keep it save.

Create a dashboard

For this example, I am just gonna create a really simple dashboard here with a random metric. Just to have something to show.

Create annotations

Next we will create some annotations to display. We are going to send them as a http post request. The URL we create annotations on is http://localhost:3000/api/annotations.

We next have to add the Authorization header. The value for this will be Bearer {your service account token we created}.

The last thing to do is send a request. The payload will look like

{
  "created": EPOCH with milliseconds, // eg 1684502867000
  "updated": EPOCH with milliseconds,
  "time": EPOCH with milliseconds,
  "timeEnd": EPOCH with milliseconds,
  "text": "Deployment", // text to display on the annotation pop up
  "tags": [ // tags to display on the annotation pop up
    "kind:deployment",
    "project:your-project-or-app-name",
    "env:prod",
    "region:eu-west-1"
  ]
}

The response should be

{
    "id": {annotation_id},
    "message": "Annotation added"
}

Now that we have created an annotation, it is time to set up our dashboard to view them.

Set up dashboard annotations

Go to the dashboard you created previously and go to it's settings.

Next click on Annotations and then Add annotation query.

Chose Grafana as the data source.

Next under Filter by, choose Tags and then turn on Match any.

You can then filter by tags you sent when sending the http request. I am just gonna filter on one tag kind:deployment.

Click Apply and Save dashboard.

Browse back to your dashboard and you now should see an annotation being displayed

If you don't see any annotations check the following things

• You are using a recent timestamp for your EPOCH time when creating your annotations
• You are filtering by the correct tags
• Your dashboard has annotations turned on

Adding to Azure DevOps pipeline

I have created a gist that can be added to your Azure DevOps pipeline.