Background Queries

Background Queries let you run long-running queries asynchronously using DataPrime or Lucene syntax. They are ideal for recurring, extensive analytical tasks (such as monthly or quarterly reports) and run in the background while you continue active monitoring within the Coralogix platform.

Use Background Queries to:

• Query logs and spans regardless of priority or time frame.
• Take advantage of larger scan and byte-size limits for in-depth data analysis.
• Lower costs by storing data in Monitoring and Compliance priority tiers, while retaining query capabilities for exploration.

Build a query

Step 1. In your Coralogix toolbar, navigate to Data Flow, then Background Queries v1.

Step 2. Select Background Query.

Step 3. Define a new query:

• Name. The query name must consist only of lowercase letters, with optional numbers and underscores. It cannot begin with a number or an underscore.
• Description. [optional] Enter a description of your query for internal purposes.
• Query. Enter a DataPrime query (recommended) or a Lucene query. DataPrime allows you to query both logs and spans simultaneously, in addition to grouping, counting, aggregating, and making other powerful transformations to your data.
• Filters.
  • Select the time range for your query.
  • [Optional] Filter by application, subsystem, and/or severity. By default, all applications, subsystems, and severities are queried unless otherwise specified.

Step 4. Select Run Background Query. Once you have set up and run your query, a test runs to validate your setup.

Save query results as a dataset

Turn any Background Query into a durable, queryable dataset you can reuse in Explore, Custom Dashboards, and API requests, without rerunning heavy scans or exporting data.

When you enable Save results to Dataset on a Background Query, Coralogix writes the query output to a persistent system dataset named system/engine.resultsets.<QUERY_ID>. The dataset is then available everywhere you work (Explore, Custom Dashboards, and APIs), so you can reuse results, join with other sources, visualize, and automate workflows over time.

Create and save a Background Query as a dataset

1. Go to Data Flow, then Background Queries v1, and select New Background Query. Author your query in DataPrime (recommended) or Lucene.
2. In Query Details, turn on Save results to a Dataset.
3. Select Run Background Query.
4. On completion:
   • The card shows Dataset: system/engine.resultsets.<QUERY_ID>.
   • Select Go to Explore to open Explore preloaded with source system/engine.resultsets.<QUERY_ID>.

Query the saved dataset

From Explore, Custom Dashboards, or elsewhere, query any dataset with:

source <dataspace>/<dataset>

Saved Background Query outputs are stored as system datasets under system/engine.resultsets.* and queried like any other dataset. For example:

source system/engine.resultsets.FCfvKq0bMsZrZgs2hLtQS

Use source system/engine.resultsets.<QUERY_ID> anywhere you can run DataPrime.

Where result datasets appear and how to manage them

Result datasets appear under Data Flow, then Dataset Management, then System Datasets, and are treated as standard system datasets.

Toggle Write Access (ingest) and Read Access (visibility) per dataset. Changes take effect immediately and persist; disabling read makes the dataset invisible without deleting stored data.

Usage and billing

System datasets consume units from your daily quota, but system/engine.resultsets.* does not. This may be subject to change.

Manage existing queries

To view and manage your existing queries, navigate to Data Flow, then Background Queries v1. In the Background Query Management screen, you can view query status and results or take other actions.

View query status

Your query appears in the management screen with the status Querying as it is being executed.

View query results

Once the query is executed, you can view your query results in one of two formats: Logs Preview or Download TSV.

Logs Preview lets you view your logs without ever indexing your data.

Download TSV lets you download a TSV file of your query results.

Clone your query

Duplicate your current query by selecting Clone. In the new duplicated query, select Run Background Query.

Limitations

See the limitations page for a comprehensive list of limitations on Background Queries usage.

API

Coralogix offers a gRPC-style and HTTP-style Background Queries API.

Permissions

To use Background Queries, you need the following permission:

Preset	Action	Description
DataQuerying	LEGACY-ARCHIVE-QUERIES:EXECUTE	Query data from the archive

To manage the resulting system datasets (toggling Write Access for ingest and Read Access for visibility under Data Flow, then Dataset Management), you need Manage Datasets or an equivalent dataset-management permission.

Learn more about Coralogix roles and permissions.