eBPF for APM

eBPF (extended Berkeley Packet Filter) is a groundbreaking Linux kernel technology for observability, networking, and security. It allows applications like Coralogix APM to safely execute custom logic within the kernel, enabling real-time, high-fidelity metrics—without requiring manual instrumentation or code changes.

Overview

https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation Coralogix leverages the OpenTelemetry eBPF instrumentation agent to provide zero-code automatic instrumentation with minimal system overhead. This open-source agent delivers a powerful observability layer that:

Follows OTel standards
Supports programming languages that are difficult to instrument traditionally
Automatically discovers and monitors services at the cluster level
Offers deep visibility into application behavior from the kernel up

This approach enables comprehensive observability across environments without invasive setup or application modifications. For background on the underlying technology, visit the eBPF Foundation.

The OpenTelemetry eBPF instrumentation agent has been donated to the CNCF OpenTelemetry project and is now part of the official OpenTelemetry ecosystem. You can explore the code and contribute via the OpenTelemetry eBPF instrumentation repository.

Benefits

Accelerated time-to-value

Traditional instrumentation setup can be time-consuming and requires deployment effort, especially across complex or distributed systems. With eBPF, monitoring starts immediately after deployment, capturing relevant metrics and events from the kernel without the need to instrument or redeploy applications. This rapid setup accelerates time-to-value for APM tools.

Real-time, system-wide observability with minimal overhead

eBPF enables an APM solution to provide in-depth, real-time visibility across all aspects of system performance and application behavior without impacting overall system efficiency, making it a highly effective choice for modern, production-grade observability.

Unrestricted monitoring for legacy and closed-source applications

For certain applications, especially legacy systems or proprietary software, adding instrumentation via OpenTelemetry or other SDKs may not be possible. eBPF shines here, as it attaches directly to system calls or network events at the kernel level, capturing relevant data regardless of application type, language, or compatibility with instrumentation libraries. This capability is crucial in environments with heterogeneous services or closed-source applications, allowing APM solutions to gather necessary performance data for uninstrumented services without any dependency on application-level telemetry support.

Security

eBPF runs securely within the kernel, with limited permissions needed. It minimizes attack surface and doesn't require changes to applications.

Vendor-neutral and open source

The OpenTelemetry eBPF instrumentation agent is vendor-neutral and open source, ensuring no vendor lock-in and allowing for community-driven development and improvements.

Concepts

DaemonSet – Deploying as a DaemonSet ensures that each Kubernetes node has an eBPF-based monitoring agent, giving full visibility across the cluster. This method scales easily as your cluster grows. Learn more about Kubernetes DaemonSets.
Automatic service discovery – The OpenTelemetry eBPF agent detects new pods, containers, and services automatically, adapting to changes in real-time without manual configuration.
Network observability – eBPF captures data on network latency, packet drops, and errors, supporting microservices monitoring with insights into inter-service communication.
Service name - The eBPF agent uses Kubernetes metadata such as deployment, daemonset, or statefulset names as the service name, enriching observability data with container orchestration context.

How it works

The OpenTelemetry eBPF instrumentation agent runs as a DaemonSet across your Kubernetes cluster, providing complete visibility into every packet transmitted or received by each service with no sampling or guesswork required; it captures everything at the kernel level.

The eBPF agent categorizes traffic by protocol, either by detecting it from raw data or identifying the originating library. Service-to-service connections are fully mapped, enabling complete interaction tracing, such as HTTP requests, gRPC calls, SQL queries, and their responses. Each interaction is enriched with OpenTelemetry (OTel) conventions, enhancing the contextual insights.

Your OpenTelemetry collector then gathers, processes, and enriches this data as metrics or spans, before sending it to Coralogix. This process provides deep visibility into service behaviors and interactions.

The figure below illustrates the data flow within the eBPF deployment.

Instrumentation support

The OpenTelemetry eBPF instrumentation agent supports:

Architecture: AMD64, ARM64
Kernel version: 5.8 and above (with BTF enabled), or RHEL 4.18 kernels build 348+
Protocols and technologies:
- HTTP/HTTPS/HTTP2: All programming languages
- gRPC: All programming languages
- SQL: MySQL, MariaDB, PostgreSQL, and other database protocols
- Redis: Redis protocol support
- Kafka: All Kafka implementations
- SSL/TLS: Support for encrypted protocols using OpenSSL

Go-specific instrumentations

For Go applications, the agent provides enhanced instrumentation for specific libraries:
Library Support
Standard Go net/http ✅
Gorilla Mux ✅
Gin ✅
gRPC-Go ✅
Go x/net/http2 ✅
Go-Redis v9 ✅
Sarama Kafka ✅
Kafka-Go ✅

APM feature matrix

The table below compares the features supported by Coralogix APM with full OpenTelemetry integration versus those of the eBPF-based APM.
Feature (K8s environments only) eBPF Full OpenTelemetry
Service catalog ✔️ ✔️
Database catalog ✔️ ✔️
Service map (only for Events2Metrics as metrics data source) ✔️ ✔️
Metrics contextualization ✔️ ✔️
Service alerts ✔️ ✔️
SLOs ✔️ ✔️
Apdex ✔️ ✔️
Span - log correlation ❌ ✔️
API error tracking ✔️ ✔️
Service - log correlation ✔️ ✔️
K8s resources correlation ✔️ ✔️
Serverless monitoring ❌ ✔️
Transactions ❌ ✔️
Spans exploration ✔️ ✔️
Traces exploration Limited ✔️
Trace map Limited ✔️
Sampling ✔️ ✔️
Span metrics ✔️ ✔️

Getting started

Prerequisites

Before deploying the OpenTelemetry eBPF instrumentation agent, ensure you have:

Kubernetes 1.24 or above
Helm 3.9 or above
Linux kernel 5.8+ with BTF enabled (or RHEL 4.18 build 348+)
eBPF enabled on the host
Privileged container capabilities (or specific Linux capabilities for unprivileged deployment)

Verify your setup:

helm version
kubectl version --short

Using the Coralogix Helm Chart

The recommended way to deploy the OpenTelemetry eBPF instrumentation agent is through the official Coralogix Helm chart.

New installation

To install the Coralogix eBPF agent deployment:

Add the Coralogix Helm repository:

helm repo add coralogix https://cgx.jfrog.io/artifactory/coralogix-charts-virtual
helm repo update

Create the required secret.

Ensure a Kubernetes secret named coralogix-keys exists and includes your Coralogix Private API Key. If it doesn't, use the command below to create it:
```
kubectl create secret generic coralogix-keys --from-literal=PRIVATE_KEY="<Your Coralogix Private API Key>"
```

Deploy the Coralogix Helm Chart with OTel eBPF:

helm upgrade --install otel-coralogix-integration coralogix/otel-integration \
--render-subchart-notes \
--set global.domain=<YOUR_DOMAIN> \
--set global.clusterName="<YOUR_CLUSTER>" \
--set opentelemetry-ebpf-instrumentation.enabled=true

Verify the deployment:

# Check if pods are running
kubectl get pods

Updating an existing Installation

To update your existing Coralogix eBPF agent deployment:

Update the Helm repository:
```
helm repo update coralogix
```
Check the current release:
```
helm list
```
Update the configuration.

Modify your values.yaml file with opentelemetry-ebpf-instrumentation.enabled=true.
Upgrade the release:

bash helm upgrade --install otel-coralogix-integration coralogix/otel-integration \ --render-subchart-notes \ -f values.yaml

Monitor the upgrade:

# Verify new pods are running
kubectl get pods

What's next? Unlock full trace capabilities

While eBPF provides excellent observability with minimal setup, adding full instrumentation to your services using OpenTelemetry unlocks advanced capabilities like trace maps, transactions, and detailed span exploration.

Instrumenting key services allows for richer insights, improved debugging, and enhanced contextualization, such as span-log correlation and stack trace visibility.

Start by identifying critical services to instrument and gradually transition for maximum observability. For guidance, consult our documentation or reach out to Support.

Additional resources

Coralogix Span Metrics

Next Aligning Coralogix and OTel Naming Conventions

Library	Support
Standard Go net/http	✅
Gorilla Mux	✅
Gin	✅
gRPC-Go	✅
Go x/net/http2	✅
Go-Redis v9	✅
Sarama Kafka	✅
Kafka-Go	✅

Feature (K8s environments only)	eBPF	Full OpenTelemetry
Service catalog	✔️	✔️
Database catalog	✔️	✔️
Service map (only for Events2Metrics as metrics data source)	✔️	✔️
Metrics contextualization	✔️	✔️
Service alerts	✔️	✔️
SLOs	✔️	✔️
Apdex	✔️	✔️
Span - log correlation	❌	✔️
API error tracking	✔️	✔️
Service - log correlation	✔️	✔️
K8s resources correlation	✔️	✔️
Serverless monitoring	❌	✔️
Transactions	❌	✔️
Spans exploration	✔️	✔️
Traces exploration	Limited	✔️
Trace map	Limited	✔️
Sampling	✔️	✔️
Span metrics	✔️	✔️