# Distributed tracing

Copy as Markdown[Open in ChatGPT](https://chatgpt.com/?q=Read%20https%3A%2F%2Fcoralogix.com%2Fdocs%2Fopentelemetry%2Finstrumentation-options%2Febpf-auto-instrumentation%2Fdistributed-tracing.md%20and%20help%20me%20with%20my%20question%20about%20this%20Coralogix%20documentation%20page.)[Open in Claude](https://claude.ai/new?q=Read%20https%3A%2F%2Fcoralogix.com%2Fdocs%2Fopentelemetry%2Finstrumentation-options%2Febpf-auto-instrumentation%2Fdistributed-tracing.md%20and%20help%20me%20with%20my%20question%20about%20this%20Coralogix%20documentation%20page.)

## Overview[​](#overview "Direct link to Overview")

OBI enables distributed tracing for applications, with certain limitations related to kernel versions and system configuration.

Tracing works by propagating the W3C `traceparent` header automatically, no manual configuration is required. OBI reads incoming trace headers, tracks execution, and injects outgoing `traceparent` headers into HTTP or gRPC requests. If the application already includes a `traceparent`, OBI uses that value instead of generating a new one. If no header is present, OBI creates a valid one following the W3C standard.

## How trace context propagation works[​](#how-trace-context-propagation-works "Direct link to How trace context propagation works")

OBI uses two primary methods for propagating trace context:

1. **Network-level injection:** Inserts trace headers at the packet level.
2. **Library-level injection for Go:** Writes directly into the application’s memory.

Depending on the language and system capabilities, OBI uses one or both methods. Some approaches rely on Linux kernel features and permissions.

Coralogix enables both trace context propagation methods by default using our helm installation. To disable context propagation, use one of the following methods:

1. Update your helm chart to use the following configuration:

   ```
   opentelemetry-ebpf-instrumentation:

     contextPropagation:

       enabled: false
   ```

2. Set the following environment variable:

   ```
   OTEL_EBPF_BPF_CONTEXT_PROPAGATION=disabled
   ```

The `disabled` and `tcp` modes are honored by Go clients as well. Header injection requires both `bpf_probe_write_user` support and a mode of `headers` or `all`; `disabled` and `tcp` keep ordinary Go tracing probes without mutating wire headers. Incoming `traceparent` parsing is unaffected in every mode, and [Go Trace API auto-instrumentation](https://coralogix.com/docs/opentelemetry/instrumentation-options/ebpf-auto-instrumentation/overview.md#go-trace-api-auto-instrumentation) stays active even with the default `disabled` mode.

These methods ensure compatibility with any OpenTelemetry-based tracing library. OBI modifies outgoing HTTP headers and, if encrypted traffic (HTTPS) is used, injects at the TCP/IP level. Note that encrypted traffic tracing only works between OBI-instrumented services and cannot pass through L7 proxies or load balancers.

* **HTTP:** Compatible with OpenTelemetry SDKs.
* **HTTPS/TLS:** Trace info added at TCP/IP level.
* **gRPC/HTTP2:** Network-level propagation via HPACK header injection through `sk_msg`, including connections whose HTTP/2 handshake happened before OBI attached.

### Supported languages for context propagation[​](#supported-languages-for-context-propagation "Direct link to Supported languages for context propagation")

* **All languages**: Network-level injection for HTTP, plus gRPC and HTTP/2 via HPACK header injection. Connections that were already open when OBI started are also covered, and Huffman-encoded incoming `traceparent` values are decoded (that decode needs Linux kernel 5.17+; on older kernels OBI falls back to its existing lookup path).
* **Go**: Memory-level injection via `bpf_probe_write_user` (HTTP, HTTPS, HTTP2, gRPC, and generic Go protocols), including generic Go TLS beyond OpenSSL. See [Go application support](#go-application-support-with-memory-injection).
* **Node.js**: Trace context propagation via async callback instrumentation.

No privileged mode is required for network-level propagation.

### Limitations for pre-existing connections[​](#limitations-for-pre-existing-connections "Direct link to Limitations for pre-existing connections")

gRPC and HTTP/2 connections that were established before OBI attached produce spans and propagate trace context. Two limits apply to those connections:

* **Method names degrade on non-Go services.** On a persistent connection, requests after the first encode the gRPC method as a compressed reference to a value that crossed the wire before OBI attached, so it can't be recovered and the span name falls back to `*`. Go services are unaffected, because OBI reads the method from process memory. Trace context itself is unaffected.
* **A shared connection stops receiving injection once a caller propagates its own context.** When OBI sees an application-supplied `traceparent` on a socket, it stops injecting on that socket entirely, including for callers on the same socket that don't send one. Injecting a second `traceparent` field would make receivers discard both, so OBI stands down instead.

Encrypted (TLS) connections are never treated as HTTP/2 for injection purposes, so OBI never writes HPACK into ciphertext.

## Kubernetes deployment guidelines[​](#kubernetes-deployment-guidelines "Direct link to Kubernetes deployment guidelines")

For Kubernetes, OBI is best deployed as a **DaemonSet** with:

* `hostNetwork: true`
* Volume mount `/sys/fs/cgroup`
* `CAP_NET_ADMIN` capability

Example deployment:

```
spec:

  serviceAccount: obi

  hostPID: true

  hostNetwork: true

  dnsPolicy: ClusterFirstWithHostNet

  containers:

  - name: obi

    resources:

      limits:

        memory: 120Mi

    terminationMessagePolicy: FallbackToLogsOnError

    image: "obi:latest"

    imagePullPolicy: "Always"

    command: [ "/obi", "--config=/config/obi-config.yml" ]

    env:

      - name: OTEL_EXPORTER_OTLP_ENDPOINT

        value: "http://otelcol:4318"

      - name: OTEL_EBPF_KUBE_METADATA_ENABLE

        value: "autodetect"

    securityContext:

      runAsUser: 0

      readOnlyRootFilesystem: true

      capabilities:

        add:

          - BPF

          - SYS_PTRACE

          - NET_RAW

          - CHECKPOINT_RESTORE

          - DAC_READ_SEARCH

          - PERFMON

          - NET_ADMIN

    volumeMounts:

      - name: cgroup

        mountPath: /sys/fs/cgroup # Required so OBI can monitor newly created sockets for outgoing requests.

      - mountPath: /config

        name: obi-config

  tolerations:

  - effect: NoSchedule

    operator: Exists

  - effect: NoExecute

    operator: Exists

  volumes:

  - name: obi-config

    configMap:

      name: obi-config

  - name: cgroup

    hostPath:

      path: /sys/fs/cgroup
```

Without the `/sys/fs/cgroup` mount, some trace context may be lost because OBI relies on this to detect new sockets and propagate context reliably.

## Kernel version requirements[​](#kernel-version-requirements "Direct link to Kernel version requirements")

For network-level context propagation, Linux kernel version **5.17+** is required. Some distributions, like RHEL 9.2, may include backported support. To bypass version checks:

```
OTEL_EBPF_OVERRIDE_BPF_LOOP_ENABLED=true
```

## Go application support with memory injection[​](#go-application-support-with-memory-injection "Direct link to Go application support with memory injection")

OBI also supports context propagation for **Go** applications using eBPF’s `bpf_probe_write_user` helper. This method enables propagation for HTTP, HTTPS, HTTP2, and gRPC, but it requires the container to:

* Run in **privileged mode** or
* Have **CAP\_SYS\_ADMIN** capability.

### Kernel lockdown mode impact[​](#kernel-lockdown-mode-impact "Direct link to Kernel lockdown mode impact")

Kernel integrity lockdown, often active with Secure Boot, blocks `bpf_probe_write_user`. OBI detects this automatically. To manually verify:

```
cat /sys/kernel/security/lockdown
```

If this shows anything other than `[none]`, memory injection won’t work.

## Container deployment for Go applications[​](#container-deployment-for-go-applications "Direct link to Container deployment for Go applications")

To enable memory-level propagation in containers, mount `/sys/kernel/security` so OBI can read kernel lockdown mode:

```
services:

  obi:

    image: coralogix/obi:latest

    environment:

      OTEL_EBPF_CONFIG_PATH: "/configs/obi-config.yml"

    volumes:

      - /sys/kernel/security:/sys/kernel/security # Needed so OBI can verify if the system is running in kernel lockdown mode.

      - /sys/fs/cgroup:/sys/fs/cgroup # Required for tracking socket creation and ensuring proper trace propagation.
```

Without these mounts, OBI assumes the kernel is not in lockdown mode. It may not function as intended for memory injection.

## Logs correlation[​](#logs-correlation "Direct link to Logs correlation")

OBI can automatically inject `trace_id` and `span_id` fields into application logs, linking logs directly to traces without code changes. JSON logs receive structured fields, and plain-text logs receive space-separated `key=value` fields by default. For full setup instructions, see [Trace-log correlation](https://coralogix.com/docs/opentelemetry/instrumentation-options/ebpf-auto-instrumentation/trace-log-correlation.md).