Create user-defined datasets in the default dataspace

Create custom datasets under the default/ dataspace to isolate log streams, prevent schema collisions, and apply dataset-level access control. Once created, route logs to specific datasets using TCO Optimizer policies, and query them directly from DataPrime or Explore.

How it works

User-defined datasets live under the default/ dataspace alongside the predefined logs and spans datasets. Data routed to a user-defined dataset:

• Is stored in your configured archive, inherited from Setup Archive.
• Is not indexed in OpenSearch, so High and Block TCO priorities are unavailable.
• Counts toward your daily unit quota at the priority of the routing policy.
• Has its usage tracked per dataset and per dataspace in Data Usage.

Streaming vs summary datasets

A user-defined dataset is created the same way regardless of its purpose. How it is populated determines its type:

• Streaming dataset — populated by a TCO Optimizer policy that routes ingested logs into the dataset in real time. Use this when you want a live, dataset-scoped feed of incoming data.
• Summary dataset — populated by Background queries v2 writing query results into the dataset. Use this for pre-aggregated or pre-filtered slices that downstream queries and dashboards can read quickly without rerunning heavy scans.

The same dataset can serve as both, but a single ingestion source is recommended per dataset to keep schema evolution predictable. Entity type and partitioning are determined by the first write, regardless of source — see the Background queries v2 entity-type and partitioning notes for details.

Create a dataset

1. Navigate to Data Flow, then Dataset Management.
2. Select the Default tab.
3. Select Create Dataset.
4. In Dataset name, enter a unique name. Names must be unique within the default/ dataspace.
5. In Description, enter an optional description.
6. In Bucket name, the storage bucket is inherited from your archive configuration. Optionally, enter a different bucket name.
7. Under Access Policy, optionally select Enabled Access Policy to restrict access.
8. Select Create.

The dataset appears in the Default tab. To query it immediately, hover over the dataset row and select to open it in Explore.

Route data to a dataset

Route logs to a user-defined dataset by setting it as the target in a TCO Optimizer policy. When creating or editing a policy, select your dataset from the Target Dataset dropdown.

TCO policies are evaluated in order. The first matching policy determines the routing target. If no policy matches, logs route to default/logs. To revert a policy to the standard logs destination, select Reset to default in the Target Dataset section.

Query a dataset

Query a user-defined dataset in DataPrime using its full dataspace path:

source default/<dataset-name>

From Explore, select your dataset from the data source selector to search and visualize it interactively.

Limitations

• Physical location is inherited from your Setup Archive and cannot be customized per dataset in this release.
• Only logs entity type data can be routed via TCO (streaming datasets). Spans cannot be streamed into a user-defined dataset. Summary datasets written by Background queries v2 support logs, spans, jsonData, and other entity types.
• High and Block TCO priority are available for default/logs only.
• Dataset names must be unique within default/ and cannot be renamed after creation.
• Deleting a dataset removes its configuration and stops routing. Data already written to storage is not deleted — retention is controlled by your S3 lifecycle policies.

Permissions

All permissions are in the dataengine permission group, under the team-datasets resource.

Action	Description
Read	Query data from the dataset
Append Data	Write new records to the dataset
Overwrite Data	Replace data in the dataset
Read Access Policy	View the dataset access policy
Manage Access Policy	Edit the dataset access policy