Troubleshooting

DataPrime is a highly sophisticated query language, which has some characteristics that must be understood. Even with this understanding, things don't behave as expected, so we've listed some of the most common issues and remediations for users of the DataPrime syntax.

"My query is running too slowly"

If your query is running slowly, then there are a few simple steps that you can take to speed things up.

Aim for key-value queries

For example filter my.key == 'hello' is orders of magnitude faster than filter my.key.contains('hello'). While both of these can perform well, complex comparisons are one of the most commomnn reasons for slower query times than expected.

Think about your timeframe

The DataPrime query engine is capable of processing days, weeks, months or even years worth of data. However, as is the case with almost all query technologies, the higher the timeframe, the more data is involved and the slower the query. Revising the time boundaries of your query will almost always speed up your query.

Minimize complex transformations

A key feature of the DataPrime syntax is transforming the data on the fly. While this is a core component of the syntax, complex transformations can negatively impact query times, due to the extra processing overhead. Where possible, aim to minimize complex transformations.

One approach that allows for the best of both worlds is to do part, or if possible, all, of the transformation using Coralogix Parsing Rules. These parsing rules happen on ingestion and they mean that you don't need to perform them in your queries, reducing query time and complexity.

Filter early

If you plan to perform a series of aggregations or complex transformations, it is best to reduce your working set to as small as possible. You can do this using the filter keyword. For example:

source logs 
| filter name == 'Chris'
| groupby path aggregate count()

This means the groupby operation only runs on documents where the name key is equal to Chris. This reduces the working set and increases query performance.

"Scan limit exceeded"

"Scan Limit Exceeded" is a warning that appears in the Coralogix UI. This is discussed at length in our Fair Usage Limits. There are some simple remediations that can be done:

Perform your query in "All Logs Mode" if you're querying logs, or "All Traces Mode" if you're querying your span data. In this mode, the query is executed against your cloud storage and the scan limit is much higher.
Aim to simplify your query. The DataPrime Query engine will constantly try to minimize the amount of data scanned. Simpler queries are easier to optimise.

For more details, as well as a full explanation of the limits, remedies and reasons for these limits, please check out our Fair Usage Limits.

"The AI Query got my query wrong"

Coralogix AI Query is powered by a Large Language Model, that is trained on the intricacies of your data. As such, your natural language queries are, in fact, prompts, used to generate DataPrime syntax.

We have a full write up of how to write the best prompts for natural language query in our Prompt Engineering document. This is the most likely cause for inaccurate query generation.

"I get a deprecated warning when I use a command"

DataPrime is a constantly changing language. We deprecate commands and functions before we remove them, and this is why you're seeing the deprecation warning.

Our Language Reference contains a detailed explanation of every function and command, as well as alternative syntax for deprecated features. If you wish to remove the deprecation warning, you can simply replace it with the more up to date solution.

Working with time in DataPrime

Next DataPrime AI query prompts