Configuring StatsD

Goal

Learn how to emit metrics with statsD.

What is StatsD

Statsd's description on its git repo is clear enough so let's just use it:

A network daemon that runs on the Node.js platform and listens for statistics, like counters and timers, sent over UDP or TCP and sends aggregates to one or more pluggable backend services (e.g., Graphite).

When to use and when not to use StatsD

You should not expect metrics generated with statsD to be 100% precise and extract business metrics from it (e.g: how many clients bought a product, or how many users registered for your newsletter) since data is sent over UDP (more on that below). Use statsD metrics to identify abnormalities in your services (e.g.: too many error statuses; methods taking too long to complete; etc).

Packages sent over User Datagram Protocol (UDP) do not have guaranteed delivery. More on that this wikipedia page.

Our current scenario and goal with statsd

Suppose we have an application which is accessed for not so many users and has an average of 10 login operations per minute. We will have a CloudWatch Alarm to notify us in case there is more than 30 unsuccessful login operations within a minute so we can find out what is going on. The application will emit events with statsD to CloudWatch using CloudWatch Agent.

Run StatsD locally

First make sure you have docker installed and running. Then run the following command:

docker run --rm\
 --name graphite\
 -p 80:80\
 -p 81:81\
 -p 2003-2004:2003-2004\
 -p 2023-2024:2023-2024\
 -p 8125:8125/udp\
 -p 8126:8126\
 hopsoft/graphite-statsd

You can already navigate to http://localhost:81 and see an empty graph from Graphite.

Our application

My application for this demo (Instructions on how to run that can be found in the README.MD of the repository.

Here is the part of the code that matters. For simplicity I kept everything in the same file:

Generate some metrics

Make some successful requests to our api:

for i in {1..30};
do
    curl -X POST -H 'Content-type: application/json' -i localhost:8080/v1/users/login -d '{"username": "firstuser", "password": "123"}'
done;

Now some requests with wrong credentials:

for i in {1..8};
do
    curl -X POST -H 'Content-type: application/json' -i localhost:8080/v1/users/login -d '{"username": "firstuser", "password": "wrong_passwd"}'
done;

Check graphite graph for the requests by navigating http://localhost:81 and open the directories in the left hand side: Metrics > stats_counts > my > custom > app > users > login and click the success and error metrics.

Change the range of time of the displayed data in the empty Graphite Composite panel by clicking the clock icon and set its Recent Time Range to 10 min.

Check your metrics