APM using OpenTelemetry Collector with Kubernetes

Coralogix now offers features of application performance monitoring (APM) for modern, cloud-native environments for those customers using OpenTelemetry collector with a Kubernetes processor. Our new features decorate all pillars of observability with additional information that extends beyond system availability, service performance, and response times.

New! Use our newly-expanded POD feature to view and compare all of the pods associated with a specific service. Compare metrics within a specific pod and across pods.

Prerequisites

1. Sign up for a Coralogix account. Set up your account on the Coralogix domain corresponding to the region within which you would like your data stored.

2. Access your Coralogix private key.

3. Install Kubernetes. This should include installation of the command-line tool kubectl, designed to operate on your Kubernetes cluster.

4. Install and configure Helm. We suggest you use this guide to familiarize yourself with the basics of using Helm to manage packages on your Kubernetes cluster.

Installation & Configuration

This particular workflow leverages Fluent Bit for logs, Open Telemetry for Traces and Prometheus for Metrics. All 3 are required to create an effective APM experience.

1. Install and configure OpenTelemetry to send your telemetry data to Coralogix. We strongly recommend configuring Otel to send Coralogix your logs, metrics, and tracing, so that you can enjoy the highest quality APM.

As part of your Otel setup, you will create a YAML-formatted override file which contains presets, built-in variables that automatically add processor and integration attributes. Among these variables are k8attributes for Kubernetes, which both link Otel-instrumented applications to Kubernetes and are a requirement for you to enjoy our new APM features.

2. Run Prometheus or make sure to set the following variables in your override file as part of your Otel setup:

• mode: Set this variable to deployment mode.
• receivers: You are required to collect prometheus metrics.

receivers:
- prometheus
- otlp

3. Install Prometheus node-exporter using this chart.

4. Install kube-state-metrics Helm Chart using this chart.

5. Run Fluent Bit agent.

APM on Your Coralogix Dashboard

Access our APM features on your Coralogix dashboard

1. In your Coralogix dashboard, click on the Explore tab > Tracing. Then click on the trace of interest.

2. Click on POD and HOST features, in addition to RELATED LOGS and SPAN LOGS. View them in either SPANS VIEW or DEPENDENCIES VIEW.

Validation

Once all the necessary component are installed and sending data, there are some steps we can take to verify that the we are receiving all the data required for APM to function.

K8s Resource Attributes and Trace Spans

To match Prometheus or Open Telemetry metrics with trace spans it is necessary to ship valid K8s Resource Attributes from Open Telemetry. Here is a screenshot of K8s Resource Attributes associated with a Span from the Coralogix Tracing UI.

The Coralogix APM backend will check for both the k8s.pod.name and k8s.node.name attributes and correlate them with metric and logs. If these values are not set or are missing, the APM process will not function correctly.

Host Metrics (Prometheus)

There are specific metrics that are required when correlating data for APM. When using Prometheus, metrics from the Kube State Metrics (KSM) and the Node Exporter are required. Note that both are included when installing the Coralogix Prometheus Operator.

• Validate the Kube State Metrics (KSM)
  
  To verify KSM metrics are being received we can run a query via the Coralogix Grafana console to validate that the KSM metric labels match the k8s.node.name attribute described in our Open Telemetry Span.

kube_node_info{node="<your node name>"}

• Validate Node Exporter Metrics
  
  The PromQL query from the previous step will also return the internal_ip label. We can use this value to verify that we are receiving Node Exporter Metrics for the relevant Node or Host.

count(node_cpu_seconds_total{instance=~"<internal_ip>.+"}) by (instance)

• Validate Pod Metrics
  
  To verify that Pod Metrics are being received from KSM we can run the following query in Grafana.

kube_pod_info{pod=~"<partialpodname>.*"}

We can then use the following queries to confirm that we are receiving Pod metrics in the correct format.

kube_pod_info{pod=~"<partialpodname>.*"}

k8s_pod_cpu_time_total{k8s_pod_name=~"<partialpodname>.*"}

These are some steps that can be taken to verify that Coralogix has everything it requires for APM. If any of the queries above return empty or invalid results, there may be an issue with the data being sent.

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].