Open in ChatGPTOpen in Claude

How to use DataPrime to combine datasets and correlate logs

Goal

By the end of this guide you should be able to:

• Use join to combine logs and traces by shared fields
• Use union to merge datasets with compatible schemas
• Use identifier-based filtering to correlate logs without a formal join

Why it matters

Real-world debugging rarely involves a single service. To understand the full picture, you often need to combine data from multiple sources—logs, traces, or metrics—based on shared identifiers like request_id, trace_id, or user_id. This guide helps you unify fragmented data into a cohesive timeline for triage, monitoring, and root cause analysis.

---

Combining datasets using join

Description

The join command combines two datasets by matching a common field (e.g., trace_id, request_id). It's useful for enriching logs with related events from another source.

Note

Joins can be resource intensive. Try to filter as much as possible before joining.

Syntax

<query1>
| join (
  <query2>
) on <join_condition>

---

Merging datasets using union

Description

The union command merges two datasets into a single stream. Both sources should have compatible schemas or be normalized with choose.

Syntax

<query1>
| union (
  <query2>
)

---

Combining data across dataspaces and tiers

join and union aren't limited to your default data. You can reference any queryable source in the same query, including:

• default/logs, default/spans, and user-defined datasets
• frequentsearch/logs and frequentsearch/spans (your High (Frequent Search) tier)
• system/* datasets such as system/engine.queries

This means you can join logs and spans wherever they sit, or union across storage tiers in a single query — for example, to compare how many records landed in Frequent Search versus your default dataset for the same timeframe.

Track where each record came from

When a query spans several datasets, use the dataspace() and dataset() functions to attribute every record to its origin, then group by both to see the breakdown. Add recordLocation() if you also need the exact storage location.

source logs
| union (source system/engine.queries)
| union (source system/dataplan.usage_events)
| create query_source.dataspace from dataspace()
| create query_source.dataset from dataset()
| groupby query_source.dataspace, query_source.dataset agg count() as count

This is especially useful once data is routed across multiple streaming datasets and you need to locate where the records for a given trace ID actually live.

---

Common pitfalls

• Unfiltered joins: Always apply filter before join to avoid performance issues.
• Mismatched schemas: Use choose to normalize fields before union.
• Missing correlation keys: Without a shared ID like request_id, correlation is not possible.