Connecting the Dots: OpenTelemetry for Beginners

If you’ve ever felt overwhelmed by words like “collector,” “instrumentation,” or “tracing,” you’re not alone. I’m right there with you 😵‍💫, still learning and piecing everything together. But as I keep connecting the dots, tools like OpenTelemetry are slowly starting to make more sense. Attending Honeycomb's workshop for OpenTelemetry really helped paint a picture of the different components and how they work together. Let me share what I’ve figured out so far in the simplest way I can, so maybe the puzzle will click for you, too.

Understanding OpenTelemetry

Think of OpenTelemetry (OTel) as a toolset for understanding what’s happening inside your software. Imagine you are organizing a large mailing process where you need to send packages to people. You want to know:

1. When each package 📦 is picked up.

2. Where it goes next ❓.

3. When it gets delivered .

Let's break down some of the components of OpenTelemetry using this analogy:

Instrumentation: Attach a tracking label to each package. This is how you collect data about each package's journey. In software, this means adding sensors to your code to collect data about what’s happening.

• Logs: Logs are like notes that track every event happening during the package's journey, like when it enters a facility or leaves for delivery. You can export logs to different backends, such as a logging service, to keep track of everything.

• Metrics: Metrics are like summaries that tell you how many packages were delivered in a day, how long deliveries took, or how many were delayed. These metrics can be exported to specialized monitoring services to analyze overall performance.

• Traces: Traces are like a detailed route map showing the complete journey of an individual package, from pickup to delivery. They can be exported to a different observability platform that helps you analyze each step of the journey in detail.

Collector: Imagine a central hub where all tracking information is gathered. The Collector collects data from all the tracking labels, processes it, and sends it for analysis. The Collector can gather data from multiple sources and send it to different places. However, it isn't always necessary, you can export data directly from your app to an observability platform.

• Receiver: The Collector uses receivers to gather data from different sources. A receiver is like the entry point for data into the Collector, allowing it to receive telemetry data from various components like your application, services, or other systems.

• Processor: Once the data is received, the processor takes over. Think of the processor as the person in the central hub who organizes, filters, and enhances the tracking information before it is sent out. Processors allow you to modify the data, such as adding additional attributes or dropping unnecessary information, ensuring that only the most relevant data moves forward.

Exporter: The Exporter sends the collected data to its final destination, like an observability platform or database. You can configure your app to export directly to a backend or use a Collector to handle the exporting.

Observability Platform: This is like a dashboard that shows the progress of all packages. It turns the collected data into insights, graphs, alerts, and dashboards, so you can understand what's happening.

Using OpenTelemetry

Currently, I am working with:

1. Azure Kubernetes Service (AKS)
2. Database: For data storage.
3. Redis: For caching.
4. Azure App Configuration: To manage application settings centrally.
5. Java with SpringBoot

We need a way to track the interactions and flow between all these components. This will help us understand how they work together and gain valuable insights, not just identify potential issues.

Starting with Azure Monitor

My journey began with the Azure Monitor instrumentation library in my Spring Boot app. This approach allowed me to auto-instrument our services and send telemetry data directly to Azure Application Insights without any code changes. For setup details, you can refer to the official Azure Monitor setup documentation.

Limitations

I wanted to add custom properties to the telemetry data and be more intentional with the data we are sending, ensuring we collect only what is necessary and meaningful.

Code Changes:
Adding custom properties was not as straightforward as I hoped. (Note: There might be a way to do this without modifying code, but I couldn’t get it to work.). For more intentional insights tailored to our business needs, manual instrumentation is required.

Vendor Lock-In: I also wanted to avoid being locked into a specific vendor.

Experimentation: I had been experimenting with OpenTelemetry on the side, specifically using auto-instrumentation for Python on one of my FastAPI services.

Switching to OpenTelemetry

Initial Goal: I wanted to avoid using a Collector and instead export telemetry data directly to Azure. The Collector adds an extra layer of setup and configuration, which feels overwhelming while I'm still trying to grasp all of this.

Limitation: The Spring OpenTelemetry auto-instrumentation library defaults to using OTLP as the exporter, and I couldn't find a straightforward way to configure it to export directly to Azure.

Azure Monitor Exporter: I thought Azure Monitor Exporter could be used to send data directly to Azure and avoid the Collector while using the auto-instrumentation Spring starter for OpenTelemetry.

Introducing the Collector: Although my initial goal was to avoid adding a Collector, I eventually introduced one to solve the challenge of exporting directly to Azure Monitor. It turned out to be more manageable than I had anticipated 😮‍💨.

Note: If anyone knows how to do this without a Collector, I'd love to know!

Deploying the Collector

OTel Helm Chart: I used the OpenTelemetry Helm chart to deploy the Collector in our AKS cluster.

Collector Contrib Image: I referenced the Collector contrib image, which includes Azure Monitor as an available exporter.

Outcome: Since we have a lot of services, the Collector gives us the ability to transform and enrich data before exporting it. Additionally, having a central place for processing data makes managing and scaling telemetry across multiple services much easier.

Setup without Collector

+----------------------------------------+
|          Instrumented Applications     |
| (e.g., FastAPI, Spring Boot with OTel) |
|----------------------------------------|
| - Generates telemetry (traces, metrics,|
|   logs) using OTel SDK                 |
| - Has its own Exporter 🚀              |
+-----------------|----------------------+
                  |
                  v
+----------------------------------------+
|         🛰️ Observability Backend       |
|----------------------------------------|
| - Data Storage                         |
| - Query & Visualization (e.g.,Azure    |
| Honeycomb, Grafana, Prometheus         |
| 📊 Insights and Monitoring             |
+----------------------------------------+

Setup with Collector

+----------------------------------------+
|          Instrumented Applications     |
| (e.g., FastAPI, Spring Boot with OTel) |
|----------------------------------------|
| - Generates telemetry (traces, metrics,|
|   logs) using OTel SDK                 |
| - Has simple Exporter to Collector 🚀  |
+-----------------|----------------------+
                  |
                  v
+----------------------------------------+
|       🚀 OpenTelemetry Collector       |
|----------------------------------------|
| 📥 Receivers                           |
|   - Collect telemetry data from apps   |
|                                        |
| 🔄 Processors                          |
|   - Transform and enrich data          |
|                                        |
| 📤 Exporters                           |
|   - Send data to observability backend |
+-----------------|----------------------+
                  |
                  v
+----------------------------------------+
|          🛰️ Observability Backend      |
|----------------------------------------|
| - Data Storage                         |
| - Query & Visualization (e.g., Azure   |
|   Honeycomb, Prometheus, Grafana)      |  
| 📊 Insights and Monitoring             |
+----------------------------------------+

Sample values.yaml for deploying collector to AKS using Helm

image:
  repository: "otel/opentelemetry-collector-contrib"
  tag: "latest"
config:
  receivers:
    otlp:
      protocols:
        http:
          endpoint: "0.0.0.0:4318"
        grpc:
          endpoint: "0.0.0.0:4317"

  processors:
    batch:
      send_batch_size: 1024

  exporters:
    azuremonitor:
      connection_string: "<your-connection-string>"

  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [batch]
        exporters: [azuremonitor]
      metrics:
        receivers: [otlp]
        processors: [batch]
        exporters: [azuremonitor]
      logs:
        receivers: [otlp]
        processors: [batch]
        exporters: [azuremonitor]
  extensions:
    health_check:
        endpoint: "0.0.0.0:13133"