DEV Community

Kim CH
Kim CH

Posted on • Edited on

Simplify observability .NET with OpenTelemetry Collector

#observability #dotnet #opentelemetry #collector

[Updated 20 July, 2024]

  • I've updated to .NET 8 & changes regarding otel collector

Overview

The Observability concept become the standard of almost system in recently. It helps team to troubleshoot what's happening inside the system. There are 3 pillars of Observability - Traces, Metrics and Logs.

Because of the various exporting ways, we have to consider one of these options when implementing

  • Support all type of exporting then toggle via settings. For example, only export to Zipkin if it's enabled
  • Or, just only export to Zipkin or Jaeger

Only one answer for the concerns

❓ Concerns

  • Is there any way that just only one export for multiple consumers? Or,
  • Is there any way that just only one export but change consumer without changing the code?

🌟 Only one answer

Objectives

  • Usability: Reasonable default configuration, supports popular protocols, runs and collects out of the box.
  • Performance: Highly stable and performant under varying loads and configurations.
  • Observability: An exemplar of an observable service.
  • Extensibility: Customizable without touching the core code.
  • Unification: Single codebase, deployable as an agent or collector with support for traces, metrics, and logs (future).
  • An image more than thousand words

Image description

💻 Let our hand dirty

👉 The below steps are just the showcase of using OTEL Collector within .NET 8. The full implementation can be found at - .NET with OpenTelemetry Collector

👉 In which, we'll export the telemetry signals from application to OTEL Collector then they'll be exported to - Zipkin or Jaeger for tracings; Prometheus; and Loki for logs

Nuget packages Directory.Packages.props 




<Project>
    <PropertyGroup>
        <ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
    </PropertyGroup>
    <ItemGroup>
        <PackageVersion Include="Grpc.AspNetCore" Version="2.64.0" />
        <PackageVersion Include="Microsoft.AspNetCore.OpenApi" Version="8.0.7" />
        <PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.9.0" />
        <PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.9.0" />
        <PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.9.0" />
        <PackageVersion Include="Serilog" Version="4.0.0" />
        <PackageVersion Include="Serilog.AspNetCore" Version="8.0.1" />
        <PackageVersion Include="Serilog.Sinks.OpenTelemetry" Version="3.0.0" />
        <PackageVersion Include="Serilog.Sinks.PeriodicBatching" Version="5.0.0" />
        <PackageVersion Include="Swashbuckle.AspNetCore" Version="6.6.2" />
    </ItemGroup>
</Project>
Enter fullscreen mode Exit fullscreen mode

Register OpenTelemetry, typically from Program.cs 




var builder = WebApplication.CreateBuilder(args);

builder.Host.AddSerilog();

builder.Services
    .AddOpenTelemetry()
    .ConfigureResource(resource => resource.AddService(observabilityOptions.ServiceName))
    .AddMetrics(observabilityOptions)
    .AddTracing(observabilityOptions);
Enter fullscreen mode Exit fullscreen mode

Configure Tracings 



private static OpenTelemetryBuilder AddTracing(this OpenTelemetryBuilder builder, ObservabilityOptions observabilityOptions)
{
    if (!observabilityOptions.EnabledTracing) return builder;

    builder.WithTracing(tracing =>
    {
        tracing
            .SetErrorStatusOnException()
            .SetSampler(new AlwaysOnSampler())
            .AddAspNetCoreInstrumentation(options =>
            {
                options.RecordException = true;
            });

        /* Add more instrument here: MassTransit, NgSql ... */

        /* ============== */
        /* Only export to OpenTelemetry collector */
        /* ============== */

        tracing
            .AddOtlpExporter(_ =>
            {
                _.Endpoint = observabilityOptions.CollectorUri;
                _.ExportProcessorType = ExportProcessorType.Batch;
                _.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.Grpc;
            });
    });

    return builder;
}
Enter fullscreen mode Exit fullscreen mode

Configure for Metrics 



private static OpenTelemetryBuilder AddMetrics(this OpenTelemetryBuilder builder, ObservabilityOptions observabilityOptions)
{
    builder.WithMetrics(metrics =>
    {
        metrics
            .AddAspNetCoreInstrumentation();

        /* Add more instrument here */

        /* ============== */
        /* Only export to OpenTelemetry collector */
        /* ============== */

        metrics
            .AddOtlpExporter(_ =>
            {
                _.Endpoint = observabilityOptions.CollectorUri;
                _.ExportProcessorType = ExportProcessorType.Batch;
                _.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.Grpc;
            });
    });

    return builder;
}
Enter fullscreen mode Exit fullscreen mode

Configure Logs 



private static WebApplicationBuilder AddSerilog(this WebApplicationBuilder builder, ObservabilityOptions observabilityOptions)
{
    var services = builder.Services;
    var configuration = builder.Configuration;

    services.AddSerilog((sp, serilog) =>
    {
        serilog
            .ReadFrom.Configuration(configuration, new ConfigurationReaderOptions
            {
                SectionName = $"{nameof(ObservabilityOptions)}:{nameof(Serilog)}"
            })
            .ReadFrom.Services(sp)
            .Enrich.FromLogContext()
            .Enrich.WithProperty("ApplicationName", observabilityOptions.ServiceName)
            .WriteTo.Console();

        /* ============== */
        /* Only export to OpenTelemetry collector */
        /* ============== */

        serilog
            .WriteTo.OpenTelemetry(c =>
            {
                c.Endpoint = observabilityOptions.CollectorUrl;
                c.Protocol = OtlpProtocol.Grpc;
                c.IncludedData = IncludedData.TraceIdField | IncludedData.SpanIdField | IncludedData.SourceContextAttribute;
                c.ResourceAttributes = new Dictionary<string, object>
                                                {
                                                    {"service.name", observabilityOptions.ServiceName},
                                                    {"index", 10},
                                                    {"flag", true},
                                                    {"value", 3.14}
                                                };
            });
    });

    return builder;
}
Enter fullscreen mode Exit fullscreen mode

The interesting here

1️⃣ - Refer to docker-compose.observability.yaml

Image description

2️⃣ - Refer to otel-collector.yaml to configure OTEL Collector



receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 1s

  resource:
    attributes:
      - action: insert
        key: loki.resource.labels
        value: service.name, service.namespace
      - action: insert
        key: loki.format
        value: json

exporters:
  debug:
    verbosity: normal

  prometheus:
    endpoint: 0.0.0.0:8889
    namespace: test-space
    resource_to_telemetry_conversion:
      enabled: true
    enable_open_metrics: true

  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

  zipkin:
    endpoint: "http://zipkin:9411/api/v2/spans"
    format: proto

  loki:
    endpoint: http://loki:3100/loki/api/v1/push
    default_labels_enabled:
      exporter: false
      job: true


extensions:
  health_check:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [debug, otlp/jaeger, zipkin]

    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, prometheus]

    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, loki]
Enter fullscreen mode Exit fullscreen mode

Let's 👍 OTEL Collector and take fully implementation at - .NET with OpenTelemetry Collector

Cheers!!! 🍻

Top comments (1)

Collapse
 
pvsunil profile image
Sunil • • Edited

Thank you for the well written article. However I am using an issue with Logging.

I am using Serilog in .Net Framework 4.8 to send logs to Opentelemetry. I am using Elastic as OpenTelemetry backend . I am able to send traces and but not logs. Please find my below code where it writes the log to the file but not sending logs to opentelemetry. Can somebody help?

    // TRACES ----- WORKING 
    _tracerProvider = Sdk.CreateTracerProviderBuilder()
   .AddAspNetInstrumentation()
   .AddHttpClientInstrumentation()
   .AddOtlpExporter(config =>
   {
       config.Endpoint = new Uri("https://abcd.es.io:443");
       config.Headers = "Authorization=ApiKey xyz";
   })
   .AddSource("SK")
   .SetResourceBuilder(
       ResourceBuilder.CreateDefault()
           .AddService(serviceName: "NLogger", serviceVersion: "1.0.0")
           .AddAttributes(resourceAttributes))
   .Build();

   var endpoint = "https://abcd.es.io:443/v1/logs";
   var protocol = OtlpProtocol.HttpProtobuf;

   // LOGGING --- NOT WORKING
   Log.Logger = new LoggerConfiguration()
   .MinimumLevel.Error()
   .WriteTo.Console()
   .WriteTo.File(@"c:\tt\1.txt")
   .WriteTo.OpenTelemetry(
   options => {
        options.Endpoint = endpoint;
        options.Protocol = protocol;
        options.IncludedData =
                IncludedData.SpanIdField
                | IncludedData.TraceIdField
                | IncludedData.MessageTemplateTextAttribute
                | IncludedData.MessageTemplateMD5HashAttribute;
        options.ResourceAttributes = new Dictionary<string, object>
        {
            ["service.name"] = "NLogger",
            ["index"] = 10,
            ["flag"] = true,
            ["pi"] = 3.14
         };
         options.Headers = new Dictionary<string, string>
         {
            ["Authorization"] = "Basic xyz", // user:abc123
         };
         options.BatchingOptions.BatchSizeLimit = 2;
         options.BatchingOptions.Period = TimeSpan.FromSeconds(2);
         options.BatchingOptions.QueueLimit = 10;
     })
     .CreateLogger();
Enter fullscreen mode Exit fullscreen mode

I tried "ApiKey" instead of "Basic" in the authorization header, but still it doesn't work. It writes the log successfully to the text file though. Please help.

Followed this page github.com/serilog/serilog-sinks-o... as reference.

Read next

greenfieldcoder profile image

Adding Revision Support to Entities in Entity Framework Core

Jimmy -

kaliumhexacyanoferrat profile image

Server Sent Events in ASP.NET Core

Andreas Nägeli -

integerman profile image

Building a Digital Dungeon Master with Semantic Kernel, C#, and Azure

Matt Eland -

zrebhi profile image

Recurring Tasks in .NET C# : All options explained

Zied Rebhi -