DEV Community

Kim CH
Kim CH

Posted on • Updated on

Simplify observability .NET with OpenTelemetry Collector

[Updated 20 July, 2024]

  • I've updated to .NET 8 & changes regarding otel collector

Overview

The Observability concept become the standard of almost system in recently. It helps team to troubleshoot what's happening inside the system. There are 3 pillars of Observability - Traces, Metrics and Logs.

Because of the various exporting ways, we have to consider one of these options when implementing

  • Support all type of exporting then toggle via settings. For example, only export to Zipkin if it's enabled
  • Or, just only export to Zipkin or Jaeger

Only one answer for the concerns

❓ Concerns

  • Is there any way that just only one export for multiple consumers? Or,
  • Is there any way that just only one export but change consumer without changing the code?

🌟 Only one answer

Objectives

  • Usability: Reasonable default configuration, supports popular protocols, runs and collects out of the box.
  • Performance: Highly stable and performant under varying loads and configurations.
  • Observability: An exemplar of an observable service.
  • Extensibility: Customizable without touching the core code.
  • Unification: Single codebase, deployable as an agent or collector with support for traces, metrics, and logs (future).
  • An image more than thousand words

Image description

πŸ’» Let our hand dirty


πŸ‘‰ The below steps are just the showcase of using OTEL Collector within .NET 8. The full implementation can be found at - .NET with OpenTelemetry Collector

πŸ‘‰ In which, we'll export the telemetry signals from application to OTEL Collector then they'll be exported to - Zipkin or Jaeger for tracings; Prometheus; and Loki for logs


Nuget packages Directory.Packages.props




<Project>
    <PropertyGroup>
        <ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
    </PropertyGroup>
    <ItemGroup>
        <PackageVersion Include="Grpc.AspNetCore" Version="2.64.0" />
        <PackageVersion Include="Microsoft.AspNetCore.OpenApi" Version="8.0.7" />
        <PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.9.0" />
        <PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.9.0" />
        <PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.9.0" />
        <PackageVersion Include="Serilog" Version="4.0.0" />
        <PackageVersion Include="Serilog.AspNetCore" Version="8.0.1" />
        <PackageVersion Include="Serilog.Sinks.OpenTelemetry" Version="3.0.0" />
        <PackageVersion Include="Serilog.Sinks.PeriodicBatching" Version="5.0.0" />
        <PackageVersion Include="Swashbuckle.AspNetCore" Version="6.6.2" />
    </ItemGroup>
</Project>



Enter fullscreen mode Exit fullscreen mode

Register OpenTelemetry, typically from Program.cs




var builder = WebApplication.CreateBuilder(args);

builder.Host.AddSerilog();

builder.Services
    .AddOpenTelemetry()
    .ConfigureResource(resource => resource.AddService(observabilityOptions.ServiceName))
    .AddMetrics(observabilityOptions)
    .AddTracing(observabilityOptions);



Enter fullscreen mode Exit fullscreen mode

Configure Tracings



private static OpenTelemetryBuilder AddTracing(this OpenTelemetryBuilder builder, ObservabilityOptions observabilityOptions)
{
    if (!observabilityOptions.EnabledTracing) return builder;

    builder.WithTracing(tracing =>
    {
        tracing
            .SetErrorStatusOnException()
            .SetSampler(new AlwaysOnSampler())
            .AddAspNetCoreInstrumentation(options =>
            {
                options.RecordException = true;
            });

        /* Add more instrument here: MassTransit, NgSql ... */

        /* ============== */
        /* Only export to OpenTelemetry collector */
        /* ============== */

        tracing
            .AddOtlpExporter(_ =>
            {
                _.Endpoint = observabilityOptions.CollectorUri;
                _.ExportProcessorType = ExportProcessorType.Batch;
                _.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.Grpc;
            });
    });

    return builder;
}


Enter fullscreen mode Exit fullscreen mode

Configure for Metrics



private static OpenTelemetryBuilder AddMetrics(this OpenTelemetryBuilder builder, ObservabilityOptions observabilityOptions)
{
    builder.WithMetrics(metrics =>
    {
        metrics
            .AddAspNetCoreInstrumentation();

        /* Add more instrument here */

        /* ============== */
        /* Only export to OpenTelemetry collector */
        /* ============== */

        metrics
            .AddOtlpExporter(_ =>
            {
                _.Endpoint = observabilityOptions.CollectorUri;
                _.ExportProcessorType = ExportProcessorType.Batch;
                _.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.Grpc;
            });
    });

    return builder;
}


Enter fullscreen mode Exit fullscreen mode

Configure Logs



private static WebApplicationBuilder AddSerilog(this WebApplicationBuilder builder, ObservabilityOptions observabilityOptions)
{
    var services = builder.Services;
    var configuration = builder.Configuration;

    services.AddSerilog((sp, serilog) =>
    {
        serilog
            .ReadFrom.Configuration(configuration, new ConfigurationReaderOptions
            {
                SectionName = $"{nameof(ObservabilityOptions)}:{nameof(Serilog)}"
            })
            .ReadFrom.Services(sp)
            .Enrich.FromLogContext()
            .Enrich.WithProperty("ApplicationName", observabilityOptions.ServiceName)
            .WriteTo.Console();

        /* ============== */
        /* Only export to OpenTelemetry collector */
        /* ============== */

        serilog
            .WriteTo.OpenTelemetry(c =>
            {
                c.Endpoint = observabilityOptions.CollectorUrl;
                c.Protocol = OtlpProtocol.Grpc;
                c.IncludedData = IncludedData.TraceIdField | IncludedData.SpanIdField | IncludedData.SourceContextAttribute;
                c.ResourceAttributes = new Dictionary<string, object>
                                                {
                                                    {"service.name", observabilityOptions.ServiceName},
                                                    {"index", 10},
                                                    {"flag", true},
                                                    {"value", 3.14}
                                                };
            });
    });

    return builder;
}


Enter fullscreen mode Exit fullscreen mode

The interesting here

1️⃣ - Refer to docker-compose.observability.yaml

Image description

2️⃣ - Refer to otel-collector.yaml to configure OTEL Collector



receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 1s

  resource:
    attributes:
      - action: insert
        key: loki.resource.labels
        value: service.name, service.namespace
      - action: insert
        key: loki.format
        value: json

exporters:
  debug:
    verbosity: normal

  prometheus:
    endpoint: 0.0.0.0:8889
    namespace: test-space
    resource_to_telemetry_conversion:
      enabled: true
    enable_open_metrics: true

  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

  zipkin:
    endpoint: "http://zipkin:9411/api/v2/spans"
    format: proto

  loki:
    endpoint: http://loki:3100/loki/api/v1/push
    default_labels_enabled:
      exporter: false
      job: true


extensions:
  health_check:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [debug, otlp/jaeger, zipkin]

    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, prometheus]

    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, loki]



Enter fullscreen mode Exit fullscreen mode

Let's πŸ‘ OTEL Collector and take fully implementation at - .NET with OpenTelemetry Collector

Cheers!!! 🍻

Top comments (1)

Collapse
 
pvsunil profile image
Sunil • Edited

Thank you for the well written article. However I am using an issue with Logging.

I am using Serilog in .Net Framework 4.8 to send logs to Opentelemetry. I am using Elastic as OpenTelemetry backend . I am able to send traces and but not logs. Please find my below code where it writes the log to the file but not sending logs to opentelemetry. Can somebody help?

    // TRACES ----- WORKING 
    _tracerProvider = Sdk.CreateTracerProviderBuilder()
   .AddAspNetInstrumentation()
   .AddHttpClientInstrumentation()
   .AddOtlpExporter(config =>
   {
       config.Endpoint = new Uri("https://abcd.es.io:443");
       config.Headers = "Authorization=ApiKey xyz";
   })
   .AddSource("SK")
   .SetResourceBuilder(
       ResourceBuilder.CreateDefault()
           .AddService(serviceName: "NLogger", serviceVersion: "1.0.0")
           .AddAttributes(resourceAttributes))
   .Build();

   var endpoint = "https://abcd.es.io:443/v1/logs";
   var protocol = OtlpProtocol.HttpProtobuf;

   // LOGGING --- NOT WORKING
   Log.Logger = new LoggerConfiguration()
   .MinimumLevel.Error()
   .WriteTo.Console()
   .WriteTo.File(@"c:\tt\1.txt")
   .WriteTo.OpenTelemetry(
   options => {
        options.Endpoint = endpoint;
        options.Protocol = protocol;
        options.IncludedData =
                IncludedData.SpanIdField
                | IncludedData.TraceIdField
                | IncludedData.MessageTemplateTextAttribute
                | IncludedData.MessageTemplateMD5HashAttribute;
        options.ResourceAttributes = new Dictionary<string, object>
        {
            ["service.name"] = "NLogger",
            ["index"] = 10,
            ["flag"] = true,
            ["pi"] = 3.14
         };
         options.Headers = new Dictionary<string, string>
         {
            ["Authorization"] = "Basic xyz", // user:abc123
         };
         options.BatchingOptions.BatchSizeLimit = 2;
         options.BatchingOptions.Period = TimeSpan.FromSeconds(2);
         options.BatchingOptions.QueueLimit = 10;
     })
     .CreateLogger();
Enter fullscreen mode Exit fullscreen mode

I tried "ApiKey" instead of "Basic" in the authorization header, but still it doesn't work. It writes the log successfully to the text file though. Please help.

Followed this page github.com/serilog/serilog-sinks-o... as reference.