DEV Community

Cover image for OTel operator: to simplify observability on kubernetes
Ashok Nagaraj
Ashok Nagaraj

Posted on

OTel operator: to simplify observability on kubernetes

OpenTelemetry is a collection of tools, APIs, and SDKs. You use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) for understanding, analyzing and debugging purposes.

OTel also promotes the concept of Bring Your Own (BYO) storage backend (eg: elastic stack, AppDynamics, Dynatrace ..) and BYO query (eg: prometheus) by providing common intermediate framework for instrumentation and data collection/transport/transform with OTLP.

OTel kubernetes operator

Image description
This operator provides 2 custom resources:

  1. OTel collector for collection/transform and transport telemetry data
  2. Auto-instrumentation through init container pattern for supported languages (Python, NodeJS and Java at the moment)

Modes of deployment

Image description
The operator(collector CR) can be deployed as:

  1. Deployment (default)
  2. DaemonSet
  3. SideCar injection Note: Multiple modes can co-exist based on the architecture (eg: side-car collectors pushing to a central Deployment aggregator) Official documentation
Installation
# Pre-req: kubernetes cluster with cert-manager enabled

# Deployment based
$ kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
namespace/opentelemetry-operator-system created
customresourcedefinition.apiextensions.k8s.io/instrumentations.opentelemetry.io created
...
deployment.apps/opentelemetry-operator-controller-manager created

$ kubectl get crds | grep opentel
instrumentations.opentelemetry.io          2022-11-14T06:14:32Z
opentelemetrycollectors.opentelemetry.io   2022-11-14T06:14:32Z

$ kubectl get deployments.apps -n opentelemetry-operator-system
NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
opentelemetry-operator-controller-manager   1/1     1            1           56s

# Create a collector like below
$ kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel-col
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
      batch:
        send_batch_size: 10000
        timeout: 10s

    exporters:
      logging:

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [logging]
EOF
opentelemetrycollector.opentelemetry.io/otel-col created
Enter fullscreen mode Exit fullscreen mode

Auto-instrumentation

  1. Create an instrumentation object
$ kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  exporter:
    endpoint: http://otel-collector:4317
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
EOF
instrumentation.opentelemetry.io/my-instrumentation created

$ kubectl get otelinst
NAME                 AGE   ENDPOINT                     SAMPLER                    SAMPLER ARG
my-instrumentation   35s   http://otel-collector:4317   parentbased_traceidratio   0.25

ku
Enter fullscreen mode Exit fullscreen mode
  1. Enabling instrumentation is done through annotations
  metadata:
    annotations:
      instrumentation.opentelemetry.io/inject-python: "true" # or inject-java or inject-nodejs
Enter fullscreen mode Exit fullscreen mode

Possible values are true, false(no instrumentation), my-instrumentation (custom instrumentation) or my-namespace/my-instrumentation (instrumentation defined in my-namespace).

Further tuning on which containers to instrument can be done with annotation instrumentation.opentelemetry.io/container-names: a,b,c...

When side-car based instrumentation is to be done, use the annotation sidecar.opentelemetry.io/inject: "true"
To use 3rd-party instrumentation, check this link


Demo

My partially working gist so-far is here

$ kubectl apply -f .
opentelemetrycollector.opentelemetry.io/sidecar-collector created
deployment.apps/flask-example created
instrumentation.opentelemetry.io/python-instrumentation created
service/flask-example created

$ kubectl get opentelemetrycollectors.opentelemetry.io,instrumentations.opentelemetry.io
NAME                                                        MODE      VERSION   AGE
opentelemetrycollector.opentelemetry.io/sidecar-collector   sidecar   0.63.1    80s

NAME                                                      AGE   ENDPOINT                SAMPLER                    SAMPLER ARG
instrumentation.opentelemetry.io/python-instrumentation   79s   http://localhost:4317   parentbased_traceidratio   0.25

$ kubectl get deployments.apps
NAME            READY   UP-TO-DATE   AVAILABLE   AGE
flask-example   1/1     1            1           49s

$ kubectl get pod flask-example-6795f8f4bf-ffhxl -o jsonpath="{.spec.containers[*].image}"
ashoka007/flask-example:0.1 ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector:0.63.1%

$ kubectl get pod flask-example-6795f8f4bf-ffhxl -o jsonpath="{.spec.containers[*].name}"
flask-example otc-container%
# otc-container is the local side-car collector
Enter fullscreen mode Exit fullscreen mode

Here is a full-fledged working demo and a detailed blog - https://isitobservable.io/open-telemetry/how-to-observe-your-kubernetes-cluster-with-opentelemetry

Top comments (0)