Managing system Dataset Availability and Ingestion
The system dataspace offers deep insight into your organization’s data structure, activity, and configuration. It lets you monitor schema changes, analyze alert history, and review audit logs to aid in troubleshooting, compliance, and operational awareness. These datasets are managed by Coralogix and they store internal metadata about your environment. Enabling it ensures you have a complete, always-available view of your system’s health and history, helping you catch issues early and make informed decisions.
While useful, these system-generated datasets may not always be needed for every team or project—and storing or exposing them unnecessarily can increase costs or surface sensitive data.
To address this, you can toggle system datasets on or off directly from the UI. This lets you control two key behaviors independently:
- Whether new data is ingested into the dataset.
- Whether the dataset is visible and queryable to users.
These settings are especially important for customers with strict retention policies, internal audit requirements, or budgetary constraints. They're also helpful for reducing cognitive load—by disabling datasets you're not using, they won’t clutter autocomplete suggestions or appear in logs queries.
How to enable or disable system datasets
You can manage your datasets from the UI by navigating to:
Data Flow > Dataset Management
Here, you can:
- View all active datasets.
- Enable or disable system datasets.
Enabling and disabling datasets
Datasets must be manually enabled before they start ingesting data. Once enabled:
- All users can query the dataset.
- The dataset counts toward your daily ingestion quota.
- Any previously written data remains accessible, even if you later disable it.
Disabling a dataset stops its ingestion, but does not delete or archive the stored data.
Understanding the two toggle controls
Each dataset has two independent settings:
| Setting | Description |
|---|---|
Write Access (Enabled/Disabled) | Controls whether new data is written to the dataset. |
Read Access (true/false) | Controls whether the dataset is visible to users (autocomplete, querying, etc.). |
Note
If Available = false, the dataset becomes invisible to users and it will be as if it doesn’t exist. However, if ingestion is still enabled, it will continue to store incoming data in the background.
This separation makes it possible to silently collect data for auditing or compliance, while keeping the dataset hidden from end-user queries.
When you update these settings, they take effect immediately and persist across page reloads. The toggles apply globally across your account and affect all users with access to dataset management.
These changes do not affect any existing stored data, only whether new data is ingested and whether the dataset is exposed to the user interface.
This feature gives teams more control over what data gets stored, when, and why—making DataPrime more cost-aware, audit-friendly, and focused on the data that actually matters.