# Distributed tracing

Copy as Markdown[Open in ChatGPT](https://chatgpt.com/?q=Read%20https%3A%2F%2Fcoralogix.com%2Fdocs%2Fopentelemetry%2Finstrumentation-options%2Febpf-auto-instrumentation%2Fdistributed-tracing.md%20and%20help%20me%20with%20my%20question%20about%20this%20Coralogix%20documentation%20page.)[Open in Claude](https://claude.ai/new?q=Read%20https%3A%2F%2Fcoralogix.com%2Fdocs%2Fopentelemetry%2Finstrumentation-options%2Febpf-auto-instrumentation%2Fdistributed-tracing.md%20and%20help%20me%20with%20my%20question%20about%20this%20Coralogix%20documentation%20page.)

## Overview[​](#overview "Direct link to Overview")

OBI enables distributed tracing for applications, with certain limitations related to kernel versions and system configuration.

Tracing works by propagating the W3C `traceparent` header automatically—no manual configuration is required. OBI reads incoming trace headers, tracks execution, and injects outgoing `traceparent` headers into HTTP or gRPC requests. If the application already includes a `traceparent`, OBI uses that value instead of generating a new one. If no header is present, OBI creates a valid one following the W3C standard.

## How trace context propagation works[​](#how-trace-context-propagation-works "Direct link to How trace context propagation works")

Changed in v0.10.0

OBI uses two primary methods for propagating trace context:

1. **Network-level injection:** Inserts trace headers at the packet level.
2. **Library-level injection for Go:** Writes directly into the application’s memory.

Depending on the language and system capabilities, OBI uses one or both methods. Some approaches rely on Linux kernel features and permissions.

Coralogix enables both trace context propagation methods by default using our helm installation. To disable context propagation, use one of the following methods:

1. Update your helm chart to use the following configuration:

   ```
   opentelemetry-ebpf-instrumentation:

     contextPropagation:

       enabled: false
   ```

2. Set the following environment variable:

   ```
   OTEL_EBPF_BPF_CONTEXT_PROPAGATION=disabled
   ```

These methods ensure compatibility with any OpenTelemetry-based tracing library. OBI modifies outgoing HTTP headers and, if encrypted traffic (HTTPS) is used, injects at the TCP/IP level. Note that encrypted traffic tracing only works between OBI-instrumented services and cannot pass through L7 proxies or load balancers.

* **HTTP:** Compatible with OpenTelemetry SDKs.
* **HTTPS/TLS:** Trace info added at TCP/IP level.
* **gRPC/HTTP2:** From OBI v0.10.0, network-level propagation is supported via HPACK header injection through `sk_msg`.

### Supported languages for context propagation[​](#supported-languages-for-context-propagation "Direct link to Supported languages for context propagation")

Changed in v0.10.0

* **All languages**: Network-level injection for HTTP. From OBI v0.10.0, network-level propagation extends to gRPC and HTTP/2 via HPACK header injection.
* **Go**: Memory-level injection via `bpf_probe_write_user` (HTTP, HTTPS, HTTP2, gRPC, and generic Go protocols). From OBI v0.9.0, generic Go TLS is supported, extending TLS visibility beyond OpenSSL to custom Go protocols. See [Go application support](#go-application-support-with-memory-injection).
* **Node.js**: Trace context propagation via async callback instrumentation.

No privileged mode is required for network-level propagation.

## Kubernetes deployment guidelines[​](#kubernetes-deployment-guidelines "Direct link to Kubernetes deployment guidelines")

For Kubernetes, OBI is best deployed as a **DaemonSet** with:

* `hostNetwork: true`
* Volume mount `/sys/fs/cgroup`
* `CAP_NET_ADMIN` capability

Example deployment:

```
spec:

  serviceAccount: obi

  hostPID: true

  hostNetwork: true

  dnsPolicy: ClusterFirstWithHostNet

  containers:

  - name: obi

    resources:

      limits:

        memory: 120Mi

    terminationMessagePolicy: FallbackToLogsOnError

    image: "obi:latest"

    imagePullPolicy: "Always"

    command: [ "/obi", "--config=/config/obi-config.yml" ]

    env:

      - name: OTEL_EXPORTER_OTLP_ENDPOINT

        value: "http://otelcol:4318"

      - name: OTEL_EBPF_KUBE_METADATA_ENABLE

        value: "autodetect"

    securityContext:

      runAsUser: 0

      readOnlyRootFilesystem: true

      capabilities:

        add:

          - BPF

          - SYS_PTRACE

          - NET_RAW

          - CHECKPOINT_RESTORE

          - DAC_READ_SEARCH

          - PERFMON

          - NET_ADMIN

    volumeMounts:

      - name: cgroup

        mountPath: /sys/fs/cgroup # Required so OBI can monitor newly created sockets for outgoing requests.

      - mountPath: /config

        name: obi-config

  tolerations:

  - effect: NoSchedule

    operator: Exists

  - effect: NoExecute

    operator: Exists

  volumes:

  - name: obi-config

    configMap:

      name: obi-config

  - name: cgroup

    hostPath:

      path: /sys/fs/cgroup
```

Without the `/sys/fs/cgroup` mount, some trace context may be lost because OBI relies on this to detect new sockets and propagate context reliably.

## Kernel version requirements[​](#kernel-version-requirements "Direct link to Kernel version requirements")

For network-level context propagation, Linux kernel version **5.17+** is required. Some distributions, like RHEL 9.2, may include backported support. To bypass version checks:

```
OTEL_EBPF_OVERRIDE_BPF_LOOP_ENABLED=true
```

## Go application support with memory injection[​](#go-application-support-with-memory-injection "Direct link to Go application support with memory injection")

OBI also supports context propagation for **Go** applications using eBPF’s `bpf_probe_write_user` helper. This method enables propagation for HTTP, HTTPS, HTTP2, and gRPC, but it requires the container to:

* Run in **privileged mode** or
* Have **CAP\_SYS\_ADMIN** capability.

### Kernel lockdown mode impact[​](#kernel-lockdown-mode-impact "Direct link to Kernel lockdown mode impact")

Kernel integrity lockdown, often active with Secure Boot, blocks `bpf_probe_write_user`. OBI detects this automatically. To manually verify:

```
cat /sys/kernel/security/lockdown
```

If this shows anything other than `[none]`, memory injection won’t work.

## Container deployment for Go applications[​](#container-deployment-for-go-applications "Direct link to Container deployment for Go applications")

To enable memory-level propagation in containers, mount `/sys/kernel/security` so OBI can read kernel lockdown mode:

```
services:

  obi:

    image: coralogix/obi:latest

    environment:

      OTEL_EBPF_CONFIG_PATH: "/configs/obi-config.yml"

    volumes:

      - /sys/kernel/security:/sys/kernel/security # Needed so OBI can verify if the system is running in kernel lockdown mode.

      - /sys/fs/cgroup:/sys/fs/cgroup # Required for tracking socket creation and ensuring proper trace propagation.
```

Without these mounts, OBI assumes the kernel is not in lockdown mode. It may not function as intended for memory injection.

## Logs correlation[​](#logs-correlation "Direct link to Logs correlation")

OBI can automatically inject `trace_id` and `span_id` fields into JSON application logs, linking logs directly to traces without code changes. For full setup instructions, see [Trace-log correlation](https://coralogix.com/docs/docs/opentelemetry/instrumentation-options/ebpf-auto-instrumentation/trace-log-correlation/.md).