Copy as Markdown[Open in ChatGPT](https://chatgpt.com/?q=Read%20https%3A%2F%2Fcoralogix.com%2Fdocs%2Fopentelemetry%2Finstrumentation-options%2Febpf-auto-instrumentation%2Ftrace-log-correlation.md%20and%20help%20me%20with%20my%20question%20about%20this%20Coralogix%20documentation%20page.)[Open in Claude](https://claude.ai/new?q=Read%20https%3A%2F%2Fcoralogix.com%2Fdocs%2Fopentelemetry%2Finstrumentation-options%2Febpf-auto-instrumentation%2Ftrace-log-correlation.md%20and%20help%20me%20with%20my%20question%20about%20this%20Coralogix%20documentation%20page.) # Trace-log correlation OBI automatically enriches application logs with trace context by injecting `trace_id` and `span_id` fields at the kernel level. This links your logs directly to distributed traces in Coralogix without requiring any code changes, enabling you to navigate from a trace span to the exact log entries produced during that operation. JSON log objects receive structured fields, and plain-text logs also receive trace context as space-separated `key=value` fields. See [Plain-text log enrichment](#plain-text-log-enrichment). ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * OBI deployed via the [Coralogix Helm chart](https://coralogix.com/docs/opentelemetry/instrumentation-options/ebpf-auto-instrumentation/getting-started.md). * Linux kernel **6.0 or later** (the log enrichment mechanism requires a `UBUF`-type `iov_iter` for overwriting user memory). * `CAP_SYS_ADMIN` capability and permission to use `bpf_probe_write_user`. * Kernel security lockdown mode set to `[none]` (verify with `cat /sys/kernel/security/lockdown`). * Target application writes logs as **JSON objects or plain text** (both are enriched). * BPF filesystem mounted at `/sys/fs/bpf`. ## How it works[​](#how-it-works "Direct link to How it works") 1. **Trace context capture**: OBI records trace IDs and span IDs during traced HTTP/gRPC operations. 2. **Log interception**: Kernel-level eBPF probes capture `write` system calls from the instrumented application. Only writes to the process's standard output and standard error are intercepted; writes to other pipes (a command substitution, application IPC, or a pipe on another file descriptor) pass through untouched and un-enriched. 3. **Field injection**: OBI adds `trace_id` and `span_id` to each log line (as JSON members in a JSON object, or as `key=value` tokens in plain text) and writes the enriched line itself. 4. **Original-line suppression**: OBI overwrites the application's original buffer with NULL bytes so the container runtime doesn't also capture the un-enriched copy. This leaves a placeholder line in the container log that your pipeline should drop. See [Filter the suppressed placeholder lines](#filter-the-suppressed-placeholder-lines). 5. **Pipeline passthrough**: The enriched logs continue through your existing log shipping pipeline (Fluent Bit, OpenTelemetry Collector, or any other forwarder) to Coralogix. For example, an application log entry like: ``` { "level": "info", "message": "Request processed", "duration_ms": 42 } ``` Becomes: ``` { "level": "info", "message": "Request processed", "duration_ms": 42, "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7" } ``` OBI only fills in fields that aren't already there. If your logger or SDK already injects trace context (for example through the Python `LoggingInstrumentor`) those values are preserved. For services that OBI detects as exporting OpenTelemetry traces themselves, only the trace field is injected, because a span ID generated by OBI wouldn't match the span the SDK emits. ## Filter the suppressed placeholder lines[​](#filter-the-suppressed-placeholder-lines "Direct link to Filter the suppressed placeholder lines") Every enriched `write` leaves one placeholder line in the container log: the application's original bytes, overwritten with NULL characters and terminated with a newline. OBI writes the enriched line separately, so suppressing the original is what stops the un-enriched duplicate from reaching your pipeline. Only standard output and standard error are enriched and suppressed, so these placeholder lines appear only on those streams; writes to other pipes are left untouched. Drop the placeholders downstream by filtering lines that match `^[\x00\s]*$`. CRI and Docker JSON log envelopes serialize NULL as the `\u0000` escape. The configurations below decode the JSON envelope before filtering, so the pattern matches real NULL bytes. * OpenTelemetry Collector * Fluent Bit * Docker JSON driver The `container` operator handles both CRI and Docker JSON formats and exposes the line in `body`: ``` receivers: filelog: include: - /var/log/pods/*/*/*.log start_at: end operators: - type: container - type: filter expr: 'body matches "^[\\x00\\s]*$"' ``` ``` [INPUT] Name tail Path /var/log/pods/*/*/*.log multiline.parser cri Tag kube.* [FILTER] Name grep Match * Exclude log ^[\x00\s]*$ ``` For the legacy Docker JSON log driver, parse the envelope first: ``` receivers: filelog: include: [/var/lib/docker/containers/*/*-json.log] operators: - type: json_parser parse_from: body parse_to: attributes - type: filter # Bracket access — `log` collides with expr-lang's math `log`. expr: 'attributes["log"] matches "^[\\x00\\s]*$"' ``` Writes larger than 8 KiB are only partially suppressed, so their leaked tail doesn't match this filter. See [Limitations](#limitations). ## Plain-text log enrichment[​](#plain-text-log-enrichment "Direct link to Plain-text log enrichment") Plain-text log lines also receive trace context, as lowercase fixed-width `key=value` tokens: ``` request failed trace_id=4bf92f3577b34da6a3ce929d0e0e4736 span_id=00f067aa0ba902b7 ``` Enabled by default. Check your log parsers before upgrading Plain-text enrichment is on by default for every service selected by the log enricher. Non-JSON writes carry the extra fields, which can break downstream parsers for structured non-JSON formats. To disable it, set `plain_text.enabled: false`. JSON enrichment is unaffected. The `key=value` format targets unstructured and free-form text. It isn't a native encoding for every structured non-JSON format, so if your logs use a strict non-JSON layout, disable plain-text enrichment rather than letting the fields land mid-record. Newline-delimited JSON is treated as structured JSON, not plain text: OBI enriches each JSON object record independently and leaves valid non-object records byte-identical. ### Plain-text configuration[​](#plain-text-configuration "Direct link to Plain-text configuration") ``` opentelemetry-ebpf-instrumentation: ebpf: log_enricher: field_names: trace_id: trace_id span_id: span_id plain_text: enabled: true placement: suffix multiline: first_line services: - service: - open_ports: '8080' ``` | Parameter | Description | Values | | ---------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | `field_names.trace_id` | Field name used for the trace ID, in both JSON and plain-text output | Default `trace_id`. Must be non-empty, distinct from `span_id`, and free of whitespace, `=`, and control characters. | | `field_names.span_id` | Field name used for the span ID, in both JSON and plain-text output | Default `span_id`. Same constraints as above. | | `plain_text.enabled` | Whether non-JSON writes receive trace context | `true` (default) or `false` | | `plain_text.placement` | Where the fields go on the line | `suffix` (default) or `prefix` | | `plain_text.multiline` | Which non-empty physical lines within one intercepted write are annotated | `first_line` (default), `last_line`, or `each_line` | Field names are used both to recognize existing trace context and to inject what's missing, so renaming them also changes which existing fields OBI treats as already populated. Empty lines and LF or CRLF line endings are preserved, and a write that isn't newline-terminated stays unterminated. OBI doesn't buffer writes or reconstruct logical multiline events across separate `write` calls, so `multiline` applies only within a single intercepted write. If you run OBI with Configuration v2, the same settings live under `extensions.obi.correlation.log_trace_annotation`. ## Runtime-specific stdout buffering[​](#runtime-specific-stdout-buffering "Direct link to Runtime-specific stdout buffering") OBI reads the trace context at the moment the `write` syscall fires. If your runtime buffers stdout and flushes asynchronously (on a different thread or after the request handler returns) the trace context is gone by the time the syscall reaches the enricher, and the log line is not enriched. | Runtime | Default stdout behavior | Works out of the box? | | ------- | ------------------------------------------------------------------------------ | ---------------------------- | | Go | `fmt.Println` calls `write` synchronously on the goroutine | Yes | | Node.js | `process.stdout.write()` is synchronous | Yes | | Java | `System.out.println()` flushes immediately by default | Yes | | Ruby | `puts` and `STDOUT.syswrite` issue `write` synchronously on the request thread | Yes | | Python | Block-buffered when stdout is not a TTY (for example, in Docker) | No. Set `PYTHONUNBUFFERED=1` | | .NET | `Console.Out` is block-buffered when stdout is a pipe | No. See [.NET](#net) | ### Python[​](#python "Direct link to Python") In Docker and Kubernetes, Python buffers stdout because it is not attached to a TTY. Set the `PYTHONUNBUFFERED=1` environment variable on the container to force line-buffered output. ### .NET[​](#net "Direct link to .NET") .NET's `Console.Out` wraps a `StreamWriter` with `AutoFlush = false`. When stdout is a pipe, writes accumulate until the buffer fills (4 KB) or the writer is flushed explicitly, at which point the `write` syscall fires from a finalizer thread or a later request that no longer carries the original trace context. Configure auto-flush at application startup: ``` var stdout = new StreamWriter(Console.OpenStandardOutput()) { AutoFlush = true }; Console.SetOut(stdout); ``` `Microsoft.Extensions.Logging.AddConsole()` (the default ASP.NET Core console logger) **does not work** even with `AutoFlush` set, because it queues entries through an internal `Channel` and drains them on a dedicated writer thread that has no trace context. Logging frameworks that work correctly: * `Console.WriteLine` with `AutoFlush = true`: synchronous on the calling thread. * Serilog `WriteTo.Console()`, synchronous by default. * NLog `targets/ColoredConsole` with `queueLimit=0`, synchronous mode. There is no .NET equivalent to Python's `PYTHONUNBUFFERED=1` environment variable. ## Enable trace-log correlation[​](#enable-trace-log-correlation "Direct link to Enable trace-log correlation") ### Step 1: Verify kernel requirements[​](#step-1-verify-kernel-requirements "Direct link to Step 1: Verify kernel requirements") Confirm your nodes meet the kernel version and capability requirements: ``` # Check kernel version (must be 6.0+) uname -r # Check lockdown mode (must be [none]) cat /sys/kernel/security/lockdown # Verify BPF filesystem mount | grep bpf ``` ### Step 2: Choose your log format[​](#step-2-choose-your-log-format "Direct link to Step 2: Choose your log format") Both JSON objects and plain text are enriched, so no format change is required. JSON is still the better target: it produces structured `trace_id` and `span_id` fields that Coralogix parses directly, whereas plain text produces `key=value` tokens that your parsing pipeline has to pick up. To output JSON, configure your logging framework accordingly: * **Python**: Use a custom `JSONFormatter` with the `logging` module. * **Go**: Use the `zap` library with its default JSON encoder. * **Java**: Use Logback with `LogstashEncoder`. * **Node.js**: Use the `pino` package. Verify your application's log output is valid JSON: ``` cat /path/to/app.log | jq empty ``` If your application emits a structured non-JSON format (for example logfmt or a fixed-column layout), decide before upgrading whether the appended `key=value` fields are safe for your parsers. See [Plain-text log enrichment](#plain-text-log-enrichment). ### Step 3: Configure the Helm chart[​](#step-3-configure-the-helm-chart "Direct link to Step 3: Configure the Helm chart") Enable trace-log correlation in your Coralogix Helm chart values: ``` opentelemetry-ebpf-instrumentation: ebpf: log_enricher: services: - service: - open_ports: '8080' ``` Replace `8080` with the port your application listens on. Add multiple entries to enrich logs from multiple services. Trace export must also be enabled (it is by default in the Coralogix Helm chart). If you have customized your configuration, verify that `otel_traces_export` has a valid endpoint: ``` opentelemetry-ebpf-instrumentation: otel_traces_export: endpoint: http://otel-collector:4318/v1/traces ``` ### Step 4: Apply the configuration[​](#step-4-apply-the-configuration "Direct link to Step 4: Apply the configuration") Upgrade your Helm release to apply the changes: ### Step 5: Verify enrichment[​](#step-5-verify-enrichment "Direct link to Step 5: Verify enrichment") After the pods restart, check your application's log output for the injected fields: ``` kubectl logs | jq 'select(.trace_id != null)' | head -5 ``` In Coralogix, navigate to **Logs Explorer** and filter for logs containing `trace_id`. Select a log entry and use the trace link to navigate directly to the associated trace in the **Spans Explorer**. ## Enricher configuration options[​](#enricher-configuration-options "Direct link to Enricher configuration options") Fine-tune the log enricher behavior using these optional parameters: | Parameter | Description | Default | | -------------------------- | ------------------------------------------- | ---------------------- | | `cache_ttl` | File descriptor cache lifetime | 30 minutes | | `cache_size` | Maximum number of cached file descriptors | Implementation-defined | | `async_writer_workers` | Number of asynchronous writer worker shards | Implementation-defined | | `async_writer_channel_len` | Queue capacity per worker shard | Implementation-defined | For the field-name and plain-text parameters, see [Plain-text configuration](#plain-text-configuration). Example with custom cache settings: ``` opentelemetry-ebpf-instrumentation: ebpf: log_enricher: cache_ttl: 15m cache_size: 1000 services: - service: - open_ports: '8080' ``` ## Limitations[​](#limitations "Direct link to Limitations") * **Per-write cap of 8 KiB**: Only the first 8 KiB of a single `write` or `writev` is enriched and suppressed. Bytes past that reach the container log un-enriched, and they don't match the placeholder filter described in [Filter the suppressed placeholder lines](#filter-the-suppressed-placeholder-lines). * **Plain-text format is fixed**: Plain-text enrichment emits space-separated `key=value` tokens only. Structured non-JSON formats that need a format-specific representation aren't supported. Disable plain-text enrichment for those services. * **No cross-write reconstruction**: The `multiline` setting selects lines within a single intercepted `write` call. OBI doesn't buffer writes or reassemble logical multiline events that span separate writes. * **Span window**: Logs are enriched only during active span windows. Logs written outside of a traced request do not receive trace context. * **Cache scope**: File descriptors are cached with a configurable TTL (default 30 minutes). Extremely short-lived processes may not benefit from caching. * **Async not supported**: Applications that use asynchronous write primitives are not yet supported. * **Synchronous writes required**: Logs must be written on the request-handling thread before the handler returns. Buffered or queued logging (for example, Python without `PYTHONUNBUFFERED`, or .NET with `Microsoft.Extensions.Logging.AddConsole()`) is not enriched. See [Runtime-specific stdout buffering](#runtime-specific-stdout-buffering). * **Kernel version**: Requires Linux kernel 6.0+, which is newer than the 5.8+ required for basic OBI functionality. ## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") ### Logs do not contain trace\_id or span\_id[​](#logs-do-not-contain-trace_id-or-span_id "Direct link to Logs do not contain trace_id or span_id") 1. Confirm kernel version is 6.0+: `uname -r`. 2. Check kernel lockdown mode: `cat /sys/kernel/security/lockdown` (must show `[none]`). 3. Verify the `log_enricher.services` section matches your application's port. 4. Ensure both trace export and log enricher are configured. 5. If your logs are JSON, verify they're valid: `cat app.log | jq empty`. A malformed object isn't enriched as JSON. 6. If your logs are plain text, confirm `plain_text.enabled` hasn't been set to `false`, and that the field names you're searching for match `field_names`. ### Plain-text fields appear where a parser does not expect them[​](#plain-text-fields-appear-where-a-parser-does-not-expect-them "Direct link to Plain-text fields appear where a parser does not expect them") If a downstream parser fails because plain-text enrichment is appending `trace_id` and `span_id` to lines it previously left alone, either move the fields with `plain_text.placement: prefix`, or set `plain_text.enabled: false` to disable enrichment for non-JSON writes. JSON enrichment is unaffected either way. ### Intermittent enrichment[​](#intermittent-enrichment "Direct link to Intermittent enrichment") If only some log entries are enriched, verify that the missing entries are written during an active traced request. Logs written outside of a traced span (for example, background tasks or startup logs) are not enriched.