Background Queries
The Background Queries feature enables you to run long-running queries asynchronously using DataPrime or Lucene syntax. Ideal for recurring, extensive analytical tasks—such as monthly or quarterly reports—this feature operates in the background, allowing you to continue active monitoring within the Coralogix platform.
Use it to:
- Seamlessly query logs and spans, regardless of priority or time frame.
- Access larger scan and byte-size limits for in-depth data analysis.
- Lower costs by storing data in Monitoring and Compliance priority tiers, while retaining query capabilities for exploration.
For immediate troubleshooting and low-latency needs, query in Explore.
Required permissions
To use this feature, you must have the following permissions:
Preset | Action | Description |
---|---|---|
DataQuerying | LEGACY-ARCHIVE-QUERIES:EXECUTE |
Query data from the archive |
Learn more about Coralogix roles and permissions here.
Build a query
STEP 1. In your Coralogix toolbar, navigate to Data Flow > Background Queries.
STEP 2. Click BACKGROUND QUERY.
STEP 3. Define a new query:
- Name. The query name must consist only of lowercase letters, with optional numbers and underscores. It cannot begin with a number or an underscore.
- Description. [optional] Enter a description of your query for internal purposes.
- Query. Enter a DataPrime query (recommended) or a Lucene query. DataPrime allows the user to query both logs and spans simultaneously, in addition to grouping, counting, aggregating, and making other powerful transformations to your data.
- Filters.
- Choose the time range for your query.
- [Optional] Filter by application, subsystem, and/or severity. By default, all applications, subsystems, and severities are queried unless otherwise specified.
STEP 4. Click RUN BACKGROUND QUERY. Once you have set up and run your query, a test will be run to validate your setup.
Manage existing queries
To view and manage your existing queries, navigate to Data Flow > Background Queries. In the Background Query Management Screen, you may view query status and results or take other actions.
View query status
Your query will appear in the Management Screen with the status “Querying” as it is being executed.
View query results
Once the query is executed, you may view your query results in one of two formats: Logs Preview or Download TSV.
Logs Preview allows you to view your logs without ever indexing your data.
The Download TSV option allows you to download a TSV file to view query results.
Clone your query
Duplicate your current query by clicking on the Clone button. In the new duplicated query, click RUN BACKGROUND QUERY.
Limitations
Background Queries are designed for extended operations and have the following limitations:
- Execution time: Queries are limited to a maximum execution time of 30 minutes, compared to 5 minutes in Explore.
- Data latency: Results may take longer to process due to the extended time ranges and larger data scans supported.
- Queued or pending queries: When a limit of 50 is reached, new queries cannot be submitted. Queries will remain queued until your team quota rises above zero.
When a limit is reached, a warning message is displayed. To prevent hitting these limits, refine your DataPrime queries using techniques outlined in this troubleshooting guide.
API
Coralogix offers a gRPC-style and HTTP-style Background Queries API.
Support
Need help?
Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.
Feel free to reach out to us via our in-app chat or by sending us an email at [email protected]