# Trace-log correlation

OBI automatically enriches JSON application logs with trace context by injecting `trace_id` and `span_id` fields at the kernel level. This links your logs directly to distributed traces in Coralogix without requiring any code changes, enabling you to navigate from a trace span to the exact log entries produced during that operation.

## Prerequisites

- OBI deployed via the [Coralogix Helm chart](https://coralogix.com/docs/opentelemetry/instrumentation-options/ebpf-auto-instrumentation/getting-started/index.md).
- Linux kernel **6.0 or later** (the log enrichment mechanism requires a `UBUF`-type `iov_iter` for overwriting user memory).
- `CAP_SYS_ADMIN` capability and permission to use `bpf_probe_write_user`.
- Kernel security lockdown mode set to `[none]` (verify with `cat /sys/kernel/security/lockdown`).
- Target application writes logs in **JSON format** — plain text logs pass through unmodified.
- BPF filesystem mounted at `/sys/fs/bpf`.

## How it works

```
flowchart LR
    A["Application<br>writes JSON log"] --> B["OBI eBPF probe<br>intercepts write syscall"]
    B --> C["Injects trace_id<br>and span_id"]
    C --> D["Enriched log<br>shipped to Coralogix"]
    D --> E["Coralogix correlates<br>logs ↔ traces"]

    class A entry
    class E success
```

1. **Trace context capture**: OBI records trace IDs and span IDs during traced HTTP/gRPC operations.
1. **Log interception**: Kernel-level eBPF probes capture `write` system calls from the instrumented application.
1. **Field injection**: OBI appends `trace_id` and `span_id` fields to each JSON log object in-place.
1. **Pipeline passthrough**: The enriched logs continue through your existing log shipping pipeline (Fluent Bit, OpenTelemetry Collector, or any other forwarder) to Coralogix.

For example, an application log entry like:

```json
{ "level": "info", "message": "Request processed", "duration_ms": 42 }
```

Becomes:

```json
{ "level": "info", "message": "Request processed", "duration_ms": 42, "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7" }
```

## Runtime-specific stdout buffering

New in v0.8.0

OBI reads the trace context at the moment the `write` syscall fires. If your runtime buffers stdout and flushes asynchronously — on a different thread or after the request handler returns — the trace context is gone by the time the syscall reaches the enricher, and the log line is not enriched.

| Runtime | Default stdout behavior                                                        | Works out of the box?         |
| ------- | ------------------------------------------------------------------------------ | ----------------------------- |
| Go      | `fmt.Println` calls `write` synchronously on the goroutine                     | Yes                           |
| Node.js | `process.stdout.write()` is synchronous                                        | Yes                           |
| Java    | `System.out.println()` flushes immediately by default                          | Yes                           |
| Ruby    | `puts` and `STDOUT.syswrite` issue `write` synchronously on the request thread | Yes                           |
| Python  | Block-buffered when stdout is not a TTY (for example, in Docker)               | No — set `PYTHONUNBUFFERED=1` |
| .NET    | `Console.Out` is block-buffered when stdout is a pipe                          | No — see [.NET](#net)         |

### Python

In Docker and Kubernetes, Python buffers stdout because it is not attached to a TTY. Set the `PYTHONUNBUFFERED=1` environment variable on the container to force line-buffered output.

### .NET

.NET's `Console.Out` wraps a `StreamWriter` with `AutoFlush = false`. When stdout is a pipe, writes accumulate until the buffer fills (4 KB) or the writer is flushed explicitly — at which point the `write` syscall fires from a finalizer thread or a later request that no longer carries the original trace context.

Configure auto-flush at application startup:

```csharp
var stdout = new StreamWriter(Console.OpenStandardOutput()) { AutoFlush = true };
Console.SetOut(stdout);
```

`Microsoft.Extensions.Logging.AddConsole()` (the default ASP.NET Core console logger) **does not work** even with `AutoFlush` set, because it queues entries through an internal `Channel` and drains them on a dedicated writer thread that has no trace context.

Logging frameworks that work correctly:

- `Console.WriteLine` with `AutoFlush = true` — synchronous on the calling thread.
- Serilog `WriteTo.Console()` — synchronous by default.
- NLog `targets/ColoredConsole` with `queueLimit=0` — synchronous mode.

There is no .NET equivalent to Python's `PYTHONUNBUFFERED=1` environment variable.

## Enable trace-log correlation

### Step 1: Verify kernel requirements

Confirm your nodes meet the kernel version and capability requirements:

```bash
# Check kernel version (must be 6.0+)
uname -r

# Check lockdown mode (must be [none])
cat /sys/kernel/security/lockdown

# Verify BPF filesystem
mount | grep bpf
```

### Step 2: Verify JSON log output

Your application must output logs in JSON format. If your application uses a logging framework, configure it for JSON output:

- **Python**: Use a custom `JSONFormatter` with the `logging` module.
- **Go**: Use the `zap` library with its default JSON encoder.
- **Java**: Use Logback with `LogstashEncoder`.
- **Node.js**: Use the `pino` package.

Verify your application's log output is valid JSON:

```bash
cat /path/to/app.log | jq empty
```

### Step 3: Configure the Helm chart

Enable trace-log correlation in your Coralogix Helm chart values:

```yaml
opentelemetry-ebpf-instrumentation:
  ebpf:
    log_enricher:
      services:
        - service:
            - open_ports: '8080'
```

Replace `8080` with the port your application listens on. Add multiple entries to enrich logs from multiple services.

Trace export must also be enabled (it is by default in the Coralogix Helm chart). If you have customized your configuration, verify that `otel_traces_export` has a valid endpoint:

```yaml
opentelemetry-ebpf-instrumentation:
  otel_traces_export:
    endpoint: http://otel-collector:4318/v1/traces
```

### Step 4: Apply the configuration

Upgrade your Helm release to apply the changes:

```bash
helm upgrade --install otel-coralogix-integration coralogix/otel-integration \
  --render-subchart-notes \
  -f values.yaml
```

### Step 5: Verify enrichment

After the pods restart, check your application's log output for the injected fields:

```bash
kubectl logs <your-app-pod> | jq 'select(.trace_id != null)' | head -5
```

In Coralogix, navigate to **Logs Explorer** and filter for logs containing `trace_id`. Select a log entry and use the trace link to navigate directly to the associated trace in the **Spans Explorer**.

## Enricher configuration options

Fine-tune the log enricher behavior using these optional parameters:

| Parameter                  | Description                                 | Default                |
| -------------------------- | ------------------------------------------- | ---------------------- |
| `cache_ttl`                | File descriptor cache lifetime              | 30 minutes             |
| `cache_size`               | Maximum number of cached file descriptors   | Implementation-defined |
| `async_writer_workers`     | Number of asynchronous writer worker shards | Implementation-defined |
| `async_writer_channel_len` | Queue capacity per worker shard             | Implementation-defined |

Example with custom cache settings:

```yaml
opentelemetry-ebpf-instrumentation:
  ebpf:
    log_enricher:
      cache_ttl: 15m
      cache_size: 1000
      services:
        - service:
            - open_ports: '8080'
```

## Limitations

- **JSON only**: Plain text logs are not modified. Ensure your application outputs structured JSON logs.
- **Span window**: Logs are enriched only during active span windows. Logs written outside of a traced request do not receive trace context.
- **Cache scope**: File descriptors are cached with a configurable TTL (default 30 minutes). Extremely short-lived processes may not benefit from caching.
- **Async not supported**: Applications that use asynchronous write primitives are not yet supported.
- **Synchronous writes required**: Logs must be written on the request-handling thread before the handler returns. Buffered or queued logging (for example, Python without `PYTHONUNBUFFERED`, or .NET with `Microsoft.Extensions.Logging.AddConsole()`) is not enriched. See [Runtime-specific stdout buffering](#runtime-specific-stdout-buffering).
- **Kernel version**: Requires Linux kernel 6.0+, which is newer than the 5.8+ required for basic OBI functionality.

## Troubleshooting

### Logs do not contain trace_id or span_id

1. Verify your logs are valid JSON: `cat app.log | jq empty`.
1. Confirm kernel version is 6.0+: `uname -r`.
1. Check kernel lockdown mode: `cat /sys/kernel/security/lockdown` (must show `[none]`).
1. Verify the `log_enricher.services` section matches your application's port.
1. Ensure both trace export and log enricher are configured.

### Intermittent enrichment

If only some log entries are enriched, verify that the missing entries are written during an active traced request. Logs written outside of a traced span (for example, background tasks or startup logs) are not enriched.