In any non-trivial application, each part of the system generates a substantial amount of data. Whatever be the domain, we want to observe, monitor, and act on this data.
We will explore visualization of real-time data in Grafana and while Prometheus is usually what is associated with in such scenarios, we will look at Redis TimeSeries.
We will create a dashboard for a simple app that generates a continuous stream of data and store it in Redis as Time Series data and visualize in Grafana, and configure alerts via Slack.
Application and Data
The client is a simple .NET console app that sends memory usage data on the Windows client, with an introduced delay of 750 milliseconds. This data includes total system memory usage (which is what is visualized) and memory used by each process.
The server is a basic Node.js app, and for simplicity it uses WebSocket connection as we have a steady stream of data being received. This data is then passed on to the primary Redis instance which will accept the incoming writes.
Note:
The client and the server apps are developed to get the data flowing in and do not follow any best practices like retries, error handling, configuration, logging etc.
In the current dev setup, the Node.js server, Redis and Grafana all run on the same Mac.
Configuring Redis
Installing Redis on Mac is straightforward following the instructions here. We have 2 Redis instances running - one to accept the writes and the other, from which Grafana will pull the data for visualization. There is significant documentation on replication and persistence in Redis and for this example, the second instance acts as a read-only replica which is replicated asynchronously.
We use the Redis TimeSeries type which, as the name suggests,¬ stores time series data. The type expects the value to be numeric and that is what we have - a series of numbers representing the memory utilization, in Megabytes, at that point in time.
For every data point, the key is the current timestamp, and the memory utilization number is the value. At this point, we have the data in memory, in the format we need, which can be visualized on a dashboard.
Note: We have a single Windows client here, but we can have multiple clients with a label for each time series data entry to query on.
Grafana and the Dashboard
Grafana is an open-source data visualization and monitoring solution - it allows us to query, visualize, set alerts and more for the data at hand. In this example, Grafana is setup to run locally on the Mac. After installation, the webpage is available at http://localhost:3000 where you can sign-in with the default credentials of admin/admin (and change the password).
First order of business in Grafana is to set up a dashboard with the data source of our choice, in this case Redis. You can create a new dashboard and provide Redis as the data source for the dashboard.
The screenshot shows the dashboard with Redis as the source and TimeSeries as the type. There are several commands available but the one we are interested is the TS.GET to fetch the data. You can also notice on the right side the numerous ways to customize the visualization with tool-tips, legend, graph styles etc.
We also apply a Transformation on the data for visualization where it creates a new data frame for each pair of data being received.
Once the entire setup is up and running, we can see the data show up in the dashboard as a continuously updated graph of memory utilization over time.
Slack Alerts for High-Memory Usage
Now that we have the dashboard setup and we can visualize the data, it would be nice if we could do something other than just looking at the graph.
As it is system memory usage, let’s say that we want to be notified if total system memory usage crosses 7000 MB on the client machine. To achieve this, an alert rule is configured where Grafana will notify on a Slack channel.
The alert also allows configuring options like how long it should evaluate the alert condition, how often to check for the condition, silencing options and more.
Here as we are using Slack, we first need to configure it to allow an incoming webhook (This webhook allows POST to the provided URL and is not shared anywhere other than in the Grafana notification setting).
Once added, you should see the notifications in Slack when the alert condition is met.
Grafana supports notification templates which can be reused across different types of notification contact points and messages can be configured and formatted too.
It also supports a whole bunch of options when it comes to alerting other than Slack.
Here are the GitHub links for the Windows client app, and the Node.js service with the Redis configs and Grafana dashboard/alert rule export.
Cover image credit: Photo by Markus Winkler on Unsplash
Top comments (0)