Distributed tracing

Open in ChatGPTOpen in Claude

Overview

OBI enables distributed tracing for applications, with certain limitations related to kernel versions and system configuration.

Tracing works by propagating the W3C traceparent header automatically—no manual configuration is required. OBI reads incoming trace headers, tracks execution, and injects outgoing traceparent headers into HTTP or gRPC requests. If the application already includes a traceparent, OBI uses that value instead of generating a new one. If no header is present, OBI creates a valid one following the W3C standard.

How trace context propagation works

Changed in v0.11.0

OBI uses two primary methods for propagating trace context:

1. Network-level injection: Inserts trace headers at the packet level.
2. Library-level injection for Go: Writes directly into the application’s memory.

Depending on the language and system capabilities, OBI uses one or both methods. Some approaches rely on Linux kernel features and permissions.

Coralogix enables both trace context propagation methods by default using our helm installation. To disable context propagation, use one of the following methods:

1. Update your helm chart to use the following configuration:

   opentelemetry-ebpf-instrumentation:
     contextPropagation:
       enabled: false

2. Set the following environment variable:

   OTEL_EBPF_BPF_CONTEXT_PROPAGATION=disabled

From OBI v0.11.0, the disabled and tcp modes are honored by Go clients as well. Earlier versions injected traceparent from Go HTTP/1, HTTP/2, and gRPC clients whenever the kernel supported bpf_probe_write_user, regardless of the configured mode — so if you set disabled or tcp before v0.11.0 and still saw injected headers on Go services, that stops after upgrading. Header injection now requires both bpf_probe_write_user support and a mode of headers or all; disabled and tcp keep ordinary Go tracing probes without mutating wire headers. Incoming traceparent parsing is unaffected in every mode, and Go Trace API auto-instrumentation stays active even with the default disabled mode.

These methods ensure compatibility with any OpenTelemetry-based tracing library. OBI modifies outgoing HTTP headers and, if encrypted traffic (HTTPS) is used, injects at the TCP/IP level. Note that encrypted traffic tracing only works between OBI-instrumented services and cannot pass through L7 proxies or load balancers.

• HTTP: Compatible with OpenTelemetry SDKs.
• HTTPS/TLS: Trace info added at TCP/IP level.
• gRPC/HTTP2: From OBI v0.10.0, network-level propagation is supported via HPACK header injection through sk_msg. From OBI v0.11.0, this also covers connections whose HTTP/2 handshake happened before OBI attached.

Supported languages for context propagation

Changed in v0.11.0

• All languages: Network-level injection for HTTP. From OBI v0.10.0, network-level propagation extends to gRPC and HTTP/2 via HPACK header injection. From OBI v0.11.0, connections that were already open when OBI started are also covered, and Huffman-encoded incoming traceparent values are decoded — that decode needs Linux kernel 5.17+, and on older kernels OBI falls back to its existing lookup path.
• Go: Memory-level injection via bpf_probe_write_user (HTTP, HTTPS, HTTP2, gRPC, and generic Go protocols). From OBI v0.9.0, generic Go TLS is supported, extending TLS visibility beyond OpenSSL to custom Go protocols. See Go application support.
• Node.js: Trace context propagation via async callback instrumentation.

No privileged mode is required for network-level propagation.

Limitations for pre-existing connections

New in v0.11.0

From OBI v0.11.0, gRPC and HTTP/2 connections that were established before OBI attached produce spans and propagate trace context. Two limits apply to those connections:

• Method names degrade on non-Go services. On a persistent connection, requests after the first encode the gRPC method as a compressed reference to a value that crossed the wire before OBI attached, so it can't be recovered and the span name falls back to *. Go services are unaffected, because OBI reads the method from process memory. Trace context itself is unaffected.
• A shared connection stops receiving injection once a caller propagates its own context. When OBI sees an application-supplied traceparent on a socket, it stops injecting on that socket entirely — including for callers on the same socket that don't send one. Injecting a second traceparent field would make receivers discard both, so OBI stands down instead.

Encrypted (TLS) connections are never treated as HTTP/2 for injection purposes, so OBI never writes HPACK into ciphertext.

Kubernetes deployment guidelines

For Kubernetes, OBI is best deployed as a DaemonSet with:

• hostNetwork: true
• Volume mount /sys/fs/cgroup
• CAP_NET_ADMIN capability

Example deployment:

spec:
  serviceAccount: obi
  hostPID: true
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  containers:
  - name: obi
    resources:
      limits:
        memory: 120Mi
    terminationMessagePolicy: FallbackToLogsOnError
    image: "obi:latest"
    imagePullPolicy: "Always"
    command: [ "/obi", "--config=/config/obi-config.yml" ]
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: "http://otelcol:4318"
      - name: OTEL_EBPF_KUBE_METADATA_ENABLE
        value: "autodetect"
    securityContext:
      runAsUser: 0
      readOnlyRootFilesystem: true
      capabilities:
        add:
          - BPF
          - SYS_PTRACE
          - NET_RAW
          - CHECKPOINT_RESTORE
          - DAC_READ_SEARCH
          - PERFMON
          - NET_ADMIN
    volumeMounts:
      - name: cgroup
        mountPath: /sys/fs/cgroup # Required so OBI can monitor newly created sockets for outgoing requests.
      - mountPath: /config
        name: obi-config
  tolerations:
  - effect: NoSchedule
    operator: Exists
  - effect: NoExecute
    operator: Exists
  volumes:
  - name: obi-config
    configMap:
      name: obi-config
  - name: cgroup
    hostPath:
      path: /sys/fs/cgroup

Without the /sys/fs/cgroup mount, some trace context may be lost because OBI relies on this to detect new sockets and propagate context reliably.

Kernel version requirements

For network-level context propagation, Linux kernel version 5.17+ is required. Some distributions, like RHEL 9.2, may include backported support. To bypass version checks:

OTEL_EBPF_OVERRIDE_BPF_LOOP_ENABLED=true

Go application support with memory injection

OBI also supports context propagation for Go applications using eBPF’s bpf_probe_write_user helper. This method enables propagation for HTTP, HTTPS, HTTP2, and gRPC, but it requires the container to:

• Run in privileged mode or
• Have CAP_SYS_ADMIN capability.

Kernel lockdown mode impact

Kernel integrity lockdown, often active with Secure Boot, blocks bpf_probe_write_user. OBI detects this automatically. To manually verify:

cat /sys/kernel/security/lockdown

If this shows anything other than [none], memory injection won’t work.

Container deployment for Go applications

To enable memory-level propagation in containers, mount /sys/kernel/security so OBI can read kernel lockdown mode:

services:
  obi:
    image: coralogix/obi:latest
    environment:
      OTEL_EBPF_CONFIG_PATH: "/configs/obi-config.yml"
    volumes:
      - /sys/kernel/security:/sys/kernel/security # Needed so OBI can verify if the system is running in kernel lockdown mode.
      - /sys/fs/cgroup:/sys/fs/cgroup # Required for tracking socket creation and ensuring proper trace propagation.

Without these mounts, OBI assumes the kernel is not in lockdown mode. It may not function as intended for memory injection.

Logs correlation

OBI can automatically inject trace_id and span_id fields into application logs, linking logs directly to traces without code changes. JSON logs receive structured fields, and from OBI v0.11.0 plain-text logs receive space-separated key=value fields by default. For full setup instructions, see Trace-log correlation.