Collecting Metrics in ASP.NET Core Applications

Monitoring and metrics are crucial for any production application. It provides visibility into how your application is performing, helps with debugging issues, and enables data-driven decisions for improving performance and scalability. This post will explore common ways to collect and expose metrics from ASP.NET Core applications.

What are Metrics?

Metrics represent quantifiable measurements about an application or system at a given time or over time. Common metrics include:

• Request Count: Number of requests received by the application
• Request Duration: How long do requests take on average to respond
• Memory Usage: Amount of memory the application is consuming
• CPU Usage: Percentage of CPU the application is using
• Exception Count: Number of exceptions thrown by the application
• Response Status Codes: Breakdown of response status codes (200, 404, 500 etc.)

A metric by itself doesn't tell much but provides insightful trends over time. Metrics are key to monitoring, alerting, performance baselining, and application capacity planning.

Built-in ASP.NET Core Metrics

ASP.NET Core comes with built-in support for collecting some basic out-of-the-box metrics. These metrics are exposed through the HttpMetrics Middleware registered by default in CreateWebHostBuilder().

Some of the key built-in metrics are:

• Request Count
• Response Count (broken down by status code)
• Total Request Duration
• Request Duration (broken down by status code) The middleware writes these metrics to the HTTP response headers on every request. It makes them easily accessible without additional infrastructure.

For example, you could view the metrics using Postman or curl to inspect the X-aspnet-Version X-AspNet-Variables-RequestCount response headers.

X-AspNet-Version: 5.0.0-rc.2.20475.5
X-AspNet-Variables-RequestCount: 6
X-AspNet-Variables-RequestDuration-Sum: 3.424999999402344
...

While convenient, exposing metrics only via headers has limitations. Headers do not support deeper historical aggregation or flexible export options required for robust monitoring. For that, additional libraries are needed.

Metric Libraries

Several open-source libraries integrate nicely with ASP.NET Core to provide more advanced metric collection capabilities:

Prometheus

Prometheus is one of the most popular open-source monitoring systems. At configurable intervals, it supports scraping metrics via HTTP from targets like ASP.NET Core apps.
The Prometheus.AspNetCore.Metrics package allows exporting the built-in ASP.NET Core metrics in the Prometheus exposition format. It also adds additional .NET/ASP.NET-specific metrics like GC stats, Requests Inflight, memory usage, etc.
The metrics endpoint is added at /metrics Prometheus can scrape to collect data. This data can then be visualized, alerted on, etc, using the Prometheus ecosystem of tools like Grafana.

StatsD

StatsD is a protocol and server for aggregating and summarizing metrics. It supports sending metrics over UDP.
The MicroSteam.StatsD library provides StatsD client functionality and integration for ASP.NET Core. It allows easy incrementing counters, timing metrics, etc, from the application code and pipeline.
These metrics can then be sent to a StatsD server and visualized using tools like Grafana. StatsD also supports integration with other backends like InfluxDB, Prometheus, etc.

Application Insights

Application Insights is a comprehensive Application Performance Management (APM) service by Microsoft for web apps hosted in any environment, including on-premises.
The Microsoft.ApplicationInsights.AspNetCore package collects the built-in ASP.NET metrics and additional diagnostics and availability data and sends it to the Application Insights resource.
It provides out-of-the-box dashboards and search and powerful analytics capabilities on the portal. Application Insights is ideal for applications hosted on Azure.

App Metrics

App Metrics is a cross-platform .NET library for instrumenting applications and collecting metrics. It supports exporting metrics to various backends like Prometheus, InfluxDB, ElasticSearch, etc.
The App.Metrics.AspNetCore package provides middleware to collect ASP.NET Core metrics like request counters, durations, etc and integrates with the App Metrics instrumentation API.
It is lightweight, has minimal overhead, and provides a consistent metrics API across platforms and languages. App Metrics works well even for non-Azure-hosted applications.

Choosing the Right Library

Here are some factors to consider when choosing a metrics library:

• Deployment Platform: Application Insights is ideal for apps hosted on Azure. Prometheus for Kubernetes/Docker environments. StatsD for generic backend agnostic collection.
• Visualization Needs: If requiring out-of-the-box dashboards, Application Insights excels. Prometheus/Grafana or custom solutions on top of StatsD/App Metrics works.
• Backend Preferences: Prometheus for Prometheus and Elasticsearch. StatsD for Sumo Logic, Datadog etc. Application Insights for its backend.
• Portability: App Metrics work across platforms. Prometheus is only for HTTP targets. Application Insights tied to MS ecosystem.
• Overhead: Application Insights and Prometheus have more startup overhead than App Metrics or StatsD.
• Feature Set: Application Insights is most fully featured for APM. Prometheus for observability. StatsD/App Metrics for lightweight collection. Hence, Application Insights is easy to use for Azure-hosted apps. Prometheus for complex on-prem/containerized setups. StatsD/App Metrics for low overhead collection with flexible exporters.

Instrumenting Code

Collecting application-level metrics requires instrumenting the code in strategic places. Here are some examples:

Track Request Timing
Use the IActionFilter to track request timing:

public class RequestTimingFilter : IActionFilter 
{
  public void OnActionExecuting(ActionExecutingContext context) 
  {
    context.HttpContext.Items["Timer"] = Stopwatch.StartNew();
  }
  public void OnActionExecuted(ActionExecutedContext context)
  {
    var timer = context.HttpContext.Items["Timer"] as Stopwatch;
    timer.Stop();
    // Record request duration
    var appMetrics = context.HttpContext.RequestServices.GetRequiredService<IAppMetrics>(); 
    appMetrics.Measure.RequestDuration.Measure(timer.Elapsed); 
  }
}

Track Dependency Calls

Track timing of external dependency calls:

public async Task<IActionResult> Index() 
{
  var timer = Stopwatch.StartNew();
  // Call dependency   
  var result = await _client.GetUsers();  
  timer.Stop();
  // Record duration
  var appMetrics = HttpContext.RequestServices.GetRequiredService<IAppMetrics>();
  appMetrics.Measure.Dependency("Database").Measure(timer.Elapsed);
  return View(result);
}

Track Exceptions

Count exceptions occurring in specific code blocks:

try
{
  // Method that could throw
}
catch (Exception ex)
{
  var appInsights = HttpContext.RequestServices.GetRequiredService<Microsoft.ApplicationInsights.TelemetryClient>();
  appInsights.TrackException(ex);  
}

Instrumenting at strategic places like this provides actionable insights without many code changes.

Exposing Metrics Endpoint

Most metric libraries provide a way to expose metrics through an endpoint that can be scraped. For example:

Prometheus

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
  app.UseMetricServer();
}

StatsD

app.UseMetricsEndpoint("/stats",
  metrics => metrics.Statsd())

Exposes metrics at /stats

App Metrics

app.UseMetricApi();

Exposes metrics via Prometheus, InfluxDB endpoints

Scraping this endpoint periodically provides a consistent way to collect metrics in production. Services like Prometheus are ideal for automated scraping.

Alerting on Metrics

While metrics provide visibility, it's also important to configure alerts to be notified of issues. Metric libraries typically integrate with alerting systems:

Prometheus Alertmanager

Prometheus has an integrated Alertmanager for grouping, routing, and sending alerts. Rules can be configured using the Prometheus Query Language to fire alerts based on metric thresholds.
For example, to alert if requests are taking more than 1 second on average:

alert: SlowRequests
expr: avg(aspnet_request_duration_seconds_bucket{le="1"}) < 0.9
for: 5m

Alerts can be sent via email, PagerDuty, Slack, etc. This provides automated notifications for performance issues.

Application Insights Alerts

In Application Insights, metric alerts can be configured on dashboards, workbooks, or directly in the portal.
Custom metric queries allow setting alerts on application-specific metrics, too. Notifications go to email, SMS, ARM templates, and more.

App Metrics Alerts

App Metrics exposes Prometheus-compatible metrics so that the Alertmanager can be used. Its plugin model also supports sending alerts to external systems like Slack.

Logging and Tracing

While metrics provide a high-level overview of an application, diagnosing issues requires correlating them with logs and traces:

ASP.NET Core Logging
The logging APIs allow capturing logs across the application. Libraries like Serilog and NLog provide rich log formatting and routing.

Application Insights Tracing

Application Insights application tracing correlates logs, exceptions, and external dependencies with requests' client-side performance.

Distributed Tracing with Prometheus

Prometheus can be combined with a distributed tracing system like Jaeger to trace requests across services. Libraries like OpenTelemetry make it easy to generate and propagate distributed traces.
Bringing logs, traces, and metrics together provides deep insights needed to debug performance bottlenecks and failures across complex applications and microservices.

Performance Benchmarking

To establish performance benchmarks, tools like:

• Gatling: Load test ASP.NET Core APIs and analyze results.
• K6: Open source load testing tool with JavaScript scripting.
• Neoload: Commercial load testing tool.
• Vegeta: CLI tool for hammering URLs.

Running performance tests against baseline configurations helps determine scalability bottlenecks and capacity planning. Metrics from tests can be exported to the same backend as production for apples-to-apples comparison.

Summary
Setting up a comprehensive monitoring and metrics platform is crucial for the production ASP.NET Core apps. The built-in middleware provides a good starting point. Libraries like Prometheus, Application Insights, StatsD, and App Metrics integrate to collect deeper metrics seamlessly. Instrumentation at the right places provides actionable insights. Alerts detect issues and bring log traces together, providing full diagnostics. Performance tests help establish benchmarks for scaling. With the right approach, actionable insights can be gained without much operational overhead.