Trace-log Correlation

OBI automatically enriches JSON application logs with trace context by injecting trace_id and span_id fields at the kernel level. This links your logs directly to distributed traces in Coralogix without requiring any code changes, enabling you to navigate from a trace span to the exact log entries produced during that operation.

Prerequisites

• OBI deployed via the Coralogix Helm chart.
• Linux kernel 6.0 or later (the log enrichment mechanism requires a UBUF-type iov_iter for overwriting user memory).
• CAP_SYS_ADMIN capability and permission to use bpf_probe_write_user.
• Kernel security lockdown mode set to [none] (verify with cat /sys/kernel/security/lockdown).
• Target application writes logs in JSON format — plain text logs pass through unmodified.
• BPF filesystem mounted at /sys/fs/bpf.

How it works

flowchart LR
    A["Application<br>writes JSON log"] --> B["OBI eBPF probe<br>intercepts write syscall"]
    B --> C["Injects trace_id<br>and span_id"]
    C --> D["Enriched log<br>shipped to Coralogix"]
    D --> E["Coralogix correlates<br>logs ↔ traces"]

1. Trace context capture: OBI records trace IDs and span IDs during traced HTTP/gRPC operations.
2. Log interception: Kernel-level eBPF probes capture write system calls from the instrumented application.
3. Field injection: OBI appends trace_id and span_id fields to each JSON log object in-place.
4. Pipeline passthrough: The enriched logs continue through your existing log shipping pipeline (Fluent Bit, OpenTelemetry Collector, or any other forwarder) to Coralogix.

For example, an application log entry like:

{ "level": "info", "message": "Request processed", "duration_ms": 42 }

Becomes:

{ "level": "info", "message": "Request processed", "duration_ms": 42, "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7" }

Enable trace-log correlation

Step 1: Verify kernel requirements

Confirm your nodes meet the kernel version and capability requirements:

# Check kernel version (must be 6.0+)
uname -r

# Check lockdown mode (must be [none])
cat /sys/kernel/security/lockdown

# Verify BPF filesystem
mount | grep bpf

Step 2: Verify JSON log output

Your application must output logs in JSON format. If your application uses a logging framework, configure it for JSON output:

• Python: Use a custom JSONFormatter with the logging module.
• Go: Use the zap library with its default JSON encoder.
• Java: Use Logback with LogstashEncoder.
• Node.js: Use the pino package.

Verify your application's log output is valid JSON:

cat /path/to/app.log | jq empty

Step 3: Configure the Helm chart

Enable trace-log correlation in your Coralogix Helm chart values:

opentelemetry-ebpf-instrumentation:
  ebpf:
    log_enricher:
      services:
        - service:
            - open_ports: '8080'

Replace 8080 with the port your application listens on. Add multiple entries to enrich logs from multiple services.

Trace export must also be enabled (it is by default in the Coralogix Helm chart). If you have customized your configuration, verify that otel_traces_export has a valid endpoint:

opentelemetry-ebpf-instrumentation:
  otel_traces_export:
    endpoint: http://otel-collector:4318/v1/traces

Step 4: Apply the configuration

Upgrade your Helm release to apply the changes:

helm upgrade --install otel-coralogix-integration coralogix/otel-integration \
  --render-subchart-notes \
  -f values.yaml

Step 5: Verify enrichment

After the pods restart, check your application's log output for the injected fields:

kubectl logs <your-app-pod> | jq 'select(.trace_id != null)' | head -5

In Coralogix, navigate to Logs Explorer and filter for logs containing trace_id. Select a log entry and use the trace link to navigate directly to the associated trace in the Spans Explorer.

Enricher configuration options

Fine-tune the log enricher behavior using these optional parameters:

Parameter	Description	Default
cache_ttl	File descriptor cache lifetime	30 minutes
cache_size	Maximum number of cached file descriptors	Implementation-defined
async_writer_workers	Number of asynchronous writer worker shards	Implementation-defined
async_writer_channel_len	Queue capacity per worker shard	Implementation-defined

Example with custom cache settings:

opentelemetry-ebpf-instrumentation:
  ebpf:
    log_enricher:
      cache_ttl: 15m
      cache_size: 1000
      services:
        - service:
            - open_ports: '8080'

Limitations

• JSON only: Plain text logs are not modified. Ensure your application outputs structured JSON logs.
• Span window: Logs are enriched only during active span windows. Logs written outside of a traced request do not receive trace context.
• Cache scope: File descriptors are cached with a configurable TTL (default 30 minutes). Extremely short-lived processes may not benefit from caching.
• Async not supported: Applications that use asynchronous write primitives are not yet supported.
• Kernel version: Requires Linux kernel 6.0+, which is newer than the 5.8+ required for basic OBI functionality.

Troubleshooting

Logs do not contain trace_id or span_id

1. Verify your logs are valid JSON: cat app.log | jq empty.
2. Confirm kernel version is 6.0+: uname -r.
3. Check kernel lockdown mode: cat /sys/kernel/security/lockdown (must show [none]).
4. Verify the log_enricher.services section matches your application's port.
5. Ensure both trace export and log enricher are configured.

Intermittent enrichment

If only some log entries are enriched, verify that the missing entries are written during an active traced request. Logs written outside of a traced span (for example, background tasks or startup logs) are not enriched.