At Elotl we develop Luna, an intelligent cluster autoscaler for Kubernetes. Luna gets deployed on customers' clusters and helps scale up and down compute resources to optimize cost.
Luna operates in environments where direct access isn’t always available. To overcome the problem of diagnosis and performance monitoring we have introduced the option for customers to securely send their Luna logs and metrics to our advanced log storage appliance. This empowers us to enhance our support capabilities, providing even more effective assistance to our customers. OpenTelemetry is fast becoming the standard for collecting metrics and logs in Kubernetes environments. We opted to run the OpenTelemetry collector as a sidecar for the Luna cluster autoscaler. It will gather and send the logs from a single pod, therefore running it as a sidecar was a perfect match. Sidecar for a single pod
A sidecar is a pod that runs alongside another pod. In our case, the Luna autoscaler writes logs to files in the /logs directory. To read these logs, we needed to share the directory between the main pod and its sidecar pod.
With Kubernetes, the OpenTelemetry collector can be deployed as a daemonset with a deployment or as a sidecar. While the daemonset and deployment set-up is well-documented in the official documentation, the sidecar set-up is documented using the OpenTelemetry operator. As of the writing of this blog post, the documentation does not cover deploying the collector as a sidecar for a single pod. Add the sidecar container
For this post, the pod we want to scrape the logs from will be named my-pod. First, add the volume and mount it within the main pod, which is part of a deployment:
Next, add the OpenTelemetry collector pod. Note that we use the otel/opentelemetry-collector-contrib image because it supports reading local directories, unlike the default otel/opentelemetry-collector image:
Configuring the OpenTelemetry Collector
With the two pods running and sharing a mounted directory, we need to configure the collector to:
The collector configuration consists of four sections:
Receivers
For receivers, we use the filelog/app receiver to read data from the /logs directory:
Processors
For processors, we use the batch and resource processors. The resource processor allows adding keys with desired metadata to each log with the attributes subsection:
Exporters
For exporters, we use the otlp (OpenTelemetry Protocol Exporter) exporter to send the logs to our cloud storage service:
Service
Finally, for service, we combine all the predefined sections into a logical pipeline:
Full ConfigMap
Here's the complete ConfigMap:
With the ConfigMap ready, we can use it as a parameter for the collector via /conf/collector.conf, this file expose the ConfigMap opentelemetry-collector-config as a YAML file:
The full listing
The final deployment snippet will look like this:
With this set-up, we're able to send our logs to our log storage appliance and we're able to more effectively help our customers when they ask for help.
Author: Henry Precheur (Senior Staff Engineer, Elotl) Comments are closed.
|