Limitations
Number limits
DataPrime supports numeric values up to a maximum of 2^53 (9,007,199,254,740,992). Using larger numbers may result in precision loss or unexpected behavior in queries.
Limitations in binary precision as defined by the IEEE 754 standard may cause certain decimal values to be stored or displayed imprecisely, even if they appear simple.
Limit on fields
As Coralogix ingests your logs, it converts them into a columnar Parquet file format. This transformation improves the performance of DataPrime queries.
Because logs — even within the same dataset — often have varying structures, they must all be normalized into a consistent column-based schema. This automatic normalization process prioritizes commonly used field names. As a result, less frequent or rarely used fields might be excluded from the Parquet schema. However, this doesn't mean the data is lost, it simply wasn't included in the Parquet column structure.
Each Parquet file in Coralogix is limited to a maximum of 5,100 columns. To ensure completeness, the latest version of DataPrime also stores the full original JSON document within a special column. If a query references a field that isn’t among the 5,100 Parquet columns, DataPrime will fall back to searching the raw JSON to retrieve the data.
Max query results
The number of results returned by a query depends on the type of the query and where it's being executed.
| Query Type | Default Limit | Maximum Limit |
|---|---|---|
| Web interface queries | ||
| Direct aggregation | 2,000 | 5,000 |
| Direct raw data | N/A | 15,000 |
| Archive query | 2,000 | 30,000 |
| Background archive query | N/A | 1,000,000 |
| API queries | ||
| Direct aggregation | 2,000 | 5,000 |
| Direct raw data | 2,000 | 15,000 |
| Archive query | 2,000 | 50,000 |
| Background archive query | N/A | 1,000,000 |
Shuffle limit
A shuffle limit refers to a constraint on the amount of data that can be processed during operations that require data shuffling, such as joins or aggregations. When a query involves significant data movement—especially in joins with large datasets—exceeding this limit can trigger a warning like shuffleFileSizeLimitReachedWarning.
| Location | Max shuffle size |
|---|---|
| Explore | 1GB |
| API | 1GB |
| Background queries | 10GB |
Shuffle limit reached
This warning may appear when using joins in DataPrime and typically occurs when the data being shuffled does not fit into memory. This can happen under various conditions depending on the query design.
Effect: The result set may be incomplete or incorrect—particularly when aggregation functions are used—because not all data will be processed.
How to avoid:
- Filter your queries to reduce dataset size.
- Decrease the number of joins in the query.
- Check for cardinality explosions (e.g., many-to-many relationships).
Scanned bytes limit
The number of bytes scanned for high-tier data is limited to 100MB for for OpenSearch queries. This limitation is placed on fetching 100 MB of high-tier data. It does not limit the scanning within the database storage.
The number of bytes scanned for archive data is limited to 30% of your daily ingest.
Block limit
In Coralogix data is divided into logical units called blocks. A block is a group of log data bundled together, typically by time intervals.
Each query over archived data must load and scan one or more of these blocks to extract the requested log lines.
The BLOCK_LIMIT error occurs when your query attempts to process too many blocks in a single execution, which exceeds the system-defined cap. This can happen due to:
- Large time ranges
- Unfiltered queries
- Queries without time filters, specific field constraints, or where clauses are more likely to hit this limit
Execution time limitations
Queries are limited to a maximum execution time of 5 minutes in Explore and 30 minutes for background queries.
Latency
Results may take longer to process due to support for extended time ranges and larger data scans.