How to run the OpenTelemetry collector as a Kubernetes sidecar

At Elotl we develop Luna, an intelligent cluster autoscaler for Kubernetes. Luna gets deployed on customers' clusters and helps scale up and down compute resources to optimize cost.

Luna operates in environments where direct access isn’t always available. To overcome the problem of diagnosis and performance monitoring we have introduced the option for customers to securely send their Luna logs and metrics to our advanced log storage appliance. This empowers us to enhance our support capabilities, providing even more effective assistance to our customers.

OpenTelemetry is fast becoming the standard for collecting metrics and logs in Kubernetes environments. We opted to run the OpenTelemetry collector as a sidecar for the Luna cluster autoscaler. It will gather and send the logs from a single pod, therefore running it as a sidecar was a perfect match.

---

Sidecar for a single pod

A sidecar is a pod that runs alongside another pod. In our case, the Luna autoscaler writes logs to files in the /logs directory. To read these logs, we needed to share the directory between the main pod and its sidecar pod.
With Kubernetes, the OpenTelemetry collector can be deployed as a daemonset with a deployment or as a sidecar. While the daemonset and deployment set-up is well-documented in the official documentation, the sidecar set-up is documented using the OpenTelemetry operator. As of the writing of this blog post, the documentation does not cover deploying the collector as a sidecar for a single pod.

Add the sidecar container

For this post, the pod we want to scrape the logs from will be named my-pod. First, add the volume and mount it within the main pod, which is part of a deployment:

apiVersion: apps/v1
kind: Deployment
...
spec:
  template:
    spec:
      containers:
      - name: my-pod
        ...
      volumeMounts:
      - name: logs
        mountPath: /logs
      volumes:
      - name: logs
        emptyDir: {}
          sizeLimit: 500Mi
    

Next, add the OpenTelemetry collector pod. Note that we use the otel/opentelemetry-collector-contrib image because it supports reading local directories, unlike the default otel/opentelemetry-collector image:

...
    spec:
      containers:
        - name: my-pod
          ...
          volumeMounts:
            - name: logs
              mountPath: /logs
        - name: opentelemetry-collector
          image: otel/opentelemetry-collector-contrib:latest
          volumeMounts:
            - name: logs
              mountPath: /logs
    

Configuring the OpenTelemetry Collector

With the two pods running and sharing a mounted directory, we need to configure the collector to:

1. Gather logs from the /logs directory
   
2. Process these logs and add metadata
   
3. Upload the logs to our log storage service
   

We'll add a ConfigMap and expose it to the collector by mounting it.

...
    spec:
      containers:
        ...
        - name: opentelemetry-collector
          image: otel/opentelemetry-collector-contrib:0.96.0
          volumeMounts:
            - name: logs
              mountPath: /logs
            - name: opentelemetry-config
              mountPath: /conf
      volumes:
        - name: logs
          emptyDir:
            sizeLimit: 500Mi
        - name: opentelemetry-config
          configMap:
            name: opentelemetry-collector-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: opentelemetry-collector-config
data:
    collector.yaml: ""
    

The collector configuration consists of four sections:

1. receivers: Specifies where the logs should be read from
2. processors: Adds metadata to the logs
3. exporters: Sets the endpoint for our cloud storage service
4. service: Combines the parameters from the previous sections
   

Receivers

For receivers, we use the filelog/app receiver to read data from the /logs directory:

acollector.yaml: |
  receivers:
    filelog/app:
      include: [ /logs/* ]
    

Processors

For processors, we use the batch and resource processors. The resource processor allows adding keys with desired metadata to each log with the attributes subsection:

collector.yaml: |
  ...
  processors:
    batch:
      timeout: 10s
    resource:
      attributes:
        - key: my-metadata-key
          value: my-metadata-value
          action: insert
    

Exporters

For exporters, we use the otlp (OpenTelemetry Protocol Exporter) exporter to send the logs to our cloud storage service:

collector.yaml: |
  ...
  exporters:
    otlp:
      endpoint: "my.cloud.storage.hostname"
    

Service

Finally, for service, we combine all the predefined sections into a logical pipeline:

collector.yaml: |
  ...
  service:
    pipelines:
      logs:
        receivers: [filelog/app]
        processors: [batch, resource]
        exporters: [otlp]
    

Full ConfigMap

Here's the complete ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: opentelemetry-collector-config
data:
    collector.yaml: |
      receivers:
        filelog/app:
          include: [ /logs/* ]
      processors:
        batch:
          timeout: 10s
        resource:
          attributes:
            - key: my-metadata-key
              value: my-metadata-value
              action: insert
      exporters:
        otlp:
          endpoint: "my.cloud.storage.hostname"
      service:
        pipelines:
          logs:
            receivers: [filelog/app]
            processors: [batch, resource]
            exporters: [otlp]
    

With the ConfigMap ready, we can use it as a parameter for the collector via /conf/collector.conf, this file expose the ConfigMap opentelemetry-collector-config as a YAML file:

...
    spec:
      containers:
        ...
        - name: opentelemetry-collector
          image: otel/opentelemetry-collector-contrib:0.96.0
          args:
            - --config=/conf/collector.yaml
          volumeMounts:
            - name: logs
              mountPath: /logs
    

The full listing

The final deployment snippet will look like this:

apiVersion: apps/v1
kind: Deployment
...
spec:
  template:
    spec:
      containers:
        - name: my-pod
          volumeMounts:
            - name: logs
              mountPath: /logs
          ...
        - name: opentelemetry-collector
          image: otel/opentelemetry-collector-contrib:0.96.0
          args:
            - --config=/conf/collector.yaml
          volumeMounts:
            - name: logs
              mountPath: /logs
            - name: opentelemetry-config
              mountPath: /conf
      volumes:
        - name: logs
          emptyDir:
            sizeLimit: 500Mi
        - name: opentelemetry-config
          configMap:
            name: opentelemetry-collector-config
---

apiVersion: v1
kind: ConfigMap
metadata:
  name: opentelemetry-collector-config
data:
    collector.yaml: |
      receivers:
        filelog/app:
          include: [ /logs/* ]
      processors:
        batch:
          timeout: 10s
        resource:
          attributes:
            - key: my-metadata-key
              value: my-metadata-value
              action: insert
      exporters:
        otlp:
          endpoint: "my.cloud.storage.hostname"
      service:
        pipelines:
          logs:
            receivers: [filelog/app]
            processors: [batch, resource]
            exporters: [otlp]
    

With this set-up, we're able to send our logs to our log storage appliance and we're able to more effectively help our customers when they ask for help.

Author:
Henry Precheur (Senior Staff Engineer, Elotl)