eBPF for APM

Overview

eBPF (extended Berkeley Packet Filter) is a revolutionary technology for modern observability, networking, and security monitoring in Linux environments. It enables applications like Coralogix APM to execute custom code safely within the kernel, delivering precise, real-time metrics without requiring instrumentation or code changes.

With minimal impact on system performance, the Coralogix eBPF agent provides a robust and versatile observability layer that adheres to OpenTelemetry conventions, supports programming versions not covered by OTel, and offers comprehensive coverage of all services at the cluster level through configurable denylists and allowlists. This allows unparalleled depth in monitoring application behavior directly at the kernel level. For more, see the eBPF foundation website.

Note

eBPF for APM is currently available in beta.

Benefits

Accelerated time-to-value

Traditional instrumentation setup can be time-consuming and requires deployment effort, especially across complex or distributed systems. With eBPF, monitoring starts immediately after deployment, capturing relevant metrics and events from the kernel without the need to instrument or redeploy applications. This rapid setup accelerates time-to-value for APM tools.

Real-time, system-wide observability with minimal overhead

eBPF enables an APM solution to provide in-depth, real-time visibility across all aspects of system performance and application behavior without impacting overall system efficiency, making it a highly effective choice for modern, production-grade observability.

Unrestricted monitoring for legacy and closed-source applications

For certain applications, especially legacy systems or proprietary software, adding instrumentation via OpenTelemetry or other SDKs may not be possible. eBPF shines here, as it attaches directly to system calls or network events at the kernel level, capturing relevant data regardless of application type, language, or compatibility with instrumentation libraries. This capability is crucial in environments with heterogeneous services or closed-source applications, allowing APM solutions to gather necessary performance data for uninstrumented services without any dependency on application-level telemetry support.

Security

eBPF runs securely within the kernel, with limited permissions needed. It minimizes attack surface and doesn’t require changes to applications.

Concepts

Terminology

DaemonSet – Deploying as a DaemonSet ensures that each Kubernetes node has an eBPF-based monitoring agent, giving full visibility across the cluster. This method scales easily as your cluster grows. Learn more about Kubernetes DaemonSets.
Automatic service discovery – eBPF agents detect new pods, containers, and services automatically, adapting to changes in real-time without manual configuration.
Network observability – eBPF captures data on network latency, packet drops, and errors, supporting microservices monitoring with insights into inter-service communication.
Service name - Unlike standard OTel instrumentation, which uses spanName metadata, the eBPF agent uses the Kubernetes owner's name, such as deployment, daemonset, statefulset, as the service name.

How it works

Our advanced eBPF agent runs as a DaemonSet across your Kubernetes cluster, providing complete visibility into every packet transmitted or received by each service with no sampling or guesswork required; we capture everything.

The eBPF agent categorizes traffic by protocol, either by detecting it from raw data or identifying the originating library. Service-to-service connections are fully mapped, enabling us to trace complete interactions, such as HTTP requests, SQL queries, and their responses. Each interaction is enriched with OpenTelemetry (OTel) conventions, enhancing the contextual insights.

Our OTel collector then gathers, processes, and enriches this data as metrics or spans, before sending it to Coralogix. This process provides deep visibility into service behaviors and interactions.

The figure below illustrates the data flow within the eBPF deployment.

Supported instrumentations

Architecture: AMD, ARM
Kernel version: 5.8 and above
Protocols and technologies:
- gRPC: Go
- HTTP: All programming languages
- SQL: MySQL, MariaDB, PostgreSQL
- Kafka: All implementations
- SSL: Synchronous OpenSSL (for all supported protocols)

APM feature matrix

The table below compares the features supported by Coralogix APM with full OpenTelemetry integration versus those of the eBPF-based APM.

Feature (K8s environments only)	eBPF	Full OpenTelemetry
Service Catalog	✔️	✔️
Database Catalog	✔️	✔️
Service Map (only for Events2Metrics as metrics data source)	✔️	✔️
Metrics contextualization	✔️	✔️
Service alerts	✔️	✔️
SLOs	✔️	✔️
Apdex	✔️	✔️
Span - log correlation	❌	✔️
API error tracking	✔️	✔️
Service - log correlation	✔️	✔️
K8s resources correlation	✔️	✔️
Serverless monitoring	❌	✔️
Service flows	❌	✔️
Spans exploration	✔️	✔️
Traces exploration	❌	✔️
Trace map	❌	✔️
Sampling	✔️	✔️
Span metrics	✔️	✔️

Getting started

Integrating Kubernetes with eBPF

Installing both the eBPF agent and the OpenTelemetry collector

To ensure compatibility, verify you have at least the following versions installed:
- Kubernetes 1.24 or above.
- Helm 3.9 or above.
```
helm version
```

To create the Coralogix Helm repository, run the following command:

    helm repo add coralogix https://cgx.jfrog.io/artifactory/coralogix-charts-virtual
    helm repo update

Enable the eBPF agent by saving the following file as ebpf.yaml.
```
coralogix-ebpf-agent:
  enabled: true
```
By default, the Coralogix eBPF agent collects traffic from all services within the cluster.
- It is recommended to configure your collector to use SpanMetrics to reduce the volume of data being collected.
- In specific scenarios, you may need to include or exclude certain services. To adjust the service filtering behavior, you can modify the coralogix-ebpf-agent.ebpf_agent.sampler parameter in the yaml file. Note that only one option, either "Allow" or "Deny", can be used for filtering at a time.
```
opentelemetry-agent:
  presets:
    spanMetrics:
      enabled: true

coralogix-ebpf-agent:
  enabled: true
  ebpf_agent:
    otel:
      sampler:
      services_filter: ["carts-service", "orders-service"]
      services_filter_type: "Allow"

coralogix-ebpf-agent:
  enabled: true
  ebpf_agent:
   otel:
     sampler:
     services_filter: ["currency-service"]
     services_filter_type: "Deny"
```
Install Coralogix collector.
1. Before setting up the integration, ensure that the Kubernetes secret named coralogix-keys exists in your cluster and contains the key PRIVATE_KEY. If the secret is missing or doesn't include this key, create it by running the following kubectl command.
```
kubectl create secret generic coralogix-keys --from-literal=PRIVATE_KEY="<Your Coralogix Private API Key>"
```
2. Once you've created the secret in your Kubernetes cluster, run the following command. Be sure to replace <cluster name> with the actual name of your cluster.
```
helm upgrade --install otel-coralogix-integration coralogix/otel-integration --version=0.0.128 --render-subchart-notes --set global.domain="<Your Coralogix Region Domain>" --set global.clusterName="<cluster name>" -f <ebpf.yaml>
```

Installing eBPF agent only

Note

To reduce volume of data, It is recommended to enable Span-Metrics within your collector.
You can also use the coralogix-ebpf-agent.ebpf_agent.sampler parameter in the yaml file to include or exclude specific services. To adjust the service filtering behavior, you can modify the coralogix-ebpf-agent.ebpf_agent.sampler parameter in the yaml file. Note that only one option, either "Allow" or "Deny", can be used for filtering at a time.

If you already have an OpenTelemetry collector deployed as a daemon on its local host, you can deploy the eBPF agent and configure it to use the existing OpenTelemetry collector endpoint.

Enable the eBPF agent by saving the following file as ebpf.yaml.

global:
  domain: "-"
  clusterName: "-"

opentelemetry-agent:
  enabled: false

opentelemetry-cluster-collector:
  enabled: false

coralogix-ebpf-agent:
  enabled: true

#Optional       
coralogix-ebpf-agent:
  enabled: true
  ebpf_agent:
    otel:
      sampler:
      services_filter: ["carts-service", "orders-service"]
      services_filter_type: "Allow"

#Optional 
coralogix-ebpf-agent:
  enabled: true
  ebpf_agent:
    otel:
      sampler:
      services_filter: ["currency-service"]
      services_filter_type: "Deny"

Run the following command.

  helm repo add coralogix-charts-virtual https://cgx.jfrog.io/artifactory/coralogix-charts-virtual

  helm upgrade --install otel-coralogix-integration coralogix/otel-integration --version=0.0.128 --render-subchart-notes -f <ebpf.yaml>

If your collector is not running as a daemon on its local host, configure it to use your existing OpenTelemetry collector endpoint as follows.

Enable the eBPF agent by saving the following file as ebpf.yaml.

  global:
      domain: "-"
      clusterName: "-"

    opentelemetry-agent:
      enabled: false

    opentelemetry-cluster-collector:
      enabled: false

    coralogix-ebpf-agent:
      enabled: true
      ebpf_agent:
        otel:
          exporter:
            # configure the opentelemetry collector endpoint here
            endpoint: "-"

    #Optional  
    coralogix-ebpf-agent:
    enabled: true
    ebpf_agent:
      otel:
        sampler:
        services_filter: ["carts-service", "orders-service"]
        services_filter_type: "Allow"

    #Optional  
    coralogix-ebpf-agent:
    enabled: true
    ebpf_agent:
      otel:
        sampler:
        services_filter: ["carts-service", "orders-service"]
        services_filter_type: "Deny"

Run the following command.

  helm repo add coralogix-charts-virtual https://cgx.jfrog.io/artifactory/coralogix-charts-virtual

  helm upgrade --install otel-coralogix-integration coralogix/otel-integration --version=0.0.128 --render-subchart-notes -f <ebpf.yaml>

Avoiding duplication with instrumented services

Coralogix APM’s service catalog provides a unified view, displaying services managed by either eBPF or full OTel. However, a service cannot present a combined time series data from both the eBPF daemon and OTel instrumentation at the same time.

To control eBPF-based data collection for instrumented services, you can either denylist services (to exclude them from eBPF monitoring) or allowlist them (to ensure only those services are monitored by the eBPF agent).
To prevent eBPF-based data from being sent for instrumented services, you can denylist services (to exclude them from eBPF monitoring) or allowlist them (to make sure they are the only services covered with the eBPF agent), using the ebpf.yaml.
To transition from eBPF to OTel instrumentation:
- Instrument your services using OTel to start generating service spans.
- Stop data collection from the eBPF agent by adding the relevant services to the denylist in the ebpf.yaml file.
Note

When switching from eBPF to full OpenTelemetry, the UI will display data according to the selected method (either eBPF or full OTel instrumentation). If both methods are used, data will be retained based on your retention plan, but only one data source can be active in the UI for each service at a time.

What to do next?

Unlock full trace capabilities

While eBPF provides excellent observability with minimal setup, adding full instrumentation to your services using OpenTelemetry unlocks advanced capabilities like trace maps, service flows, and detailed span exploration.

Instrumenting key services allows for richer insights, improved debugging, and enhanced contextualization, such as span-log correlation and stack trace visibility.

Start by identifying critical services to instrument and gradually transition for maximum observability. For guidance, consult our documentation or reach out to support.

OpenTelemetry Span Metrics