Skip to content

Create user-defined datasets in the default dataspace

Note

This feature is available for early-access customers. To request access and confirm your organization meets the feature criteria, contact your account representative or Support.

Create custom datasets under the default/ dataspace to isolate log streams, prevent schema collisions, and apply dataset-level access control. Once created, route logs to specific datasets using TCO Optimizer policies, and query them directly from DataPrime or Explore.

How it works

User-defined datasets live under the default/ dataspace alongside the predefined logs and spans datasets. Data routed to a user-defined dataset:

  • Is stored in your configured archive, inherited from Setup Archive.
  • Is not indexed in OpenSearch, so High and Block TCO priorities are unavailable.
  • Counts toward your daily unit quota at the priority of the routing policy.
  • Has its usage tracked per dataset and per dataspace in Data Usage.

Create a dataset

  1. Navigate to Data Flow, then Dataset Management.
  2. Select the Default tab.
  3. Select Create Dataset.
  4. In Dataset name, enter a unique name. Names must be unique within the default/ dataspace.
  5. In Description, enter an optional description.
  6. In Bucket name, the storage bucket is inherited from your archive configuration. Optionally, enter a different bucket name.
  7. Under Access Policy, optionally select Enabled Access Policy to restrict access.
  8. Select Create.

The dataset appears in the Default tab. To query it immediately, hover over the dataset row and select to open it in Explore.

Route data to a dataset

Route logs to a user-defined dataset by setting it as the target in a TCO Optimizer policy. When creating or editing a policy, select your dataset from the Target Dataset dropdown.

TCO policies are evaluated in order. The first matching policy determines the routing target. If no policy matches, logs route to default/logs. To revert a policy to the standard logs destination, select Reset to default in the Target Dataset section.

Query a dataset

Query a user-defined dataset in DataPrime using its full dataspace path:

source default/<dataset-name>

From Explore, select your dataset from the data source selector to search and visualize it interactively.

Limitations

  • Physical location is inherited from your Setup Archive and cannot be customized per dataset in this release.
  • Only logs entity type data can be routed to user-defined datasets. Spans are not supported.
  • High and Block TCO priority are available for default/logs only.
  • Dataset names must be unique within default/ and cannot be renamed after creation.
  • Deleting a dataset removes its configuration and stops routing. Data already written to storage is not deleted — retention is controlled by your S3 lifecycle policies.

Permissions

All permissions are in the dataengine permission group, under the team-datasets resource.
ActionDescription
ReadQuery data from the dataset
Append DataWrite new records to the dataset
Overwrite DataReplace data in the dataset
Read Access PolicyView the dataset access policy
Manage Access PolicyEdit the dataset access policy

Need help? Contact Support.
What's new? Find out here.
LLM? Read llms.txt.
Previous Default dataspace
Next System dataspace