242 docs tagged with "dataengine"

The schema for aaa.audit_events is coming soon.

Returns the absolute value of a number. Useful for computing the total difference between two values without regard to sign.

System datasets support fine-grained access control that combines permissions and policies. This lets you manage dataset visibility and edit rights with precision.

This guide explains how to query your S3 Coralogix archive bucket (cx-data) using a third-party framework with the standard Apache Parquet reader provided by the framework and the required schema.

Metadata fields ($m) aren't shown as structured key/value rows in the Log details panel — they appear only nested in the panel's raw JSON. Querying them with the $m. mechanisms below is the most reliable way to read and inspect metadata values.

Problem / Use case

Returns the sum of two intervals as a single interval value. For example, adding an interval of `1d` (one day) to another `1d` produces `2d` (two days). !!! note This function also supports negative intervals. For example, `1d + -1h` results in `23h`, following standard arithmetic rules where adding a negative is equivalent to subtraction.

Returns a new timestamp by adding an interval to an existing timestamp. Both `timestamp` and `interval` are first-class types in DataPrime, so ensure that the correct types are passed. !!! note Negative intervals are supported and act as subtraction. For example, adding `-1h` to a timestamp subtracts one hour.

The `aggregate` command performs calculations across the entire working set of documents, producing summary statistics such as totals, averages, minimums, maximums, or counts. Unlike `groupby`, which splits data into multiple groups, `aggregate` computes results over the full dataset as a single group. Multiple aggregation functions can be combined in one command to produce a single document containing several computed values. !!! note Aggregations are limited to 1000 buckets in a single operation.

Goal

Schema, access model, and opt-in content capture for the ai_sessions_claude dataset that stores Claude Code prompts, responses, tool calls, and errors.

The alerts.history dataset is generated only for alerts processed through Notification Center. Customers must use Notification Center to have read and write access to this dataset.

This tutorial demonstrates configuring an Amazon S3 bucket to send your telemetry data to Coralogix. For GCP environments, see Connect a GCS archive bucket.

Returns any non-null value from the specified expression within a group. If no expression is provided, it defaults to the `$d` object. Returns `null` if all values in the group are `null`.

Run DataPrime or Lucene queries directly from your code—no need to open the Coralogix UI.

Returns an approximate count of unique, non-null values for a given expression. * A document is counted if it contains a unique, non-null value. * The result is approximate, optimized for performance rather than precision. !!! note `approx_count_distinct` is an aggregation function and must be used with a grouping keyword such as `groupby`.

Archive Query enables you to directly query your logs from your S3 archive using any text or Lucene or DataPrime syntax query. Query logs from your Explore Screen irrespective of log priority, daily quota, or the time frame of the data - all with the ease of familiar functionalities.

Control and modify the length of time your logs are archived using Archive Retention policies in Coralogix.

The `around` keyword allows users to declare a timerange, defined as some interval before and after a `timestamp`, on which a query should operate. !!! note The `interval`, if specified, **MUST** be positive. !!! note If the `interval` is not specified, then a default value of `30m` is used.

Returns a new array with an additional element appended to the end of an existing array. * The element type must match the array type. * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`.

Returns a new array by concatenating the elements of two arrays into a single array. * Both arrays must be of the same element type. * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`.

Returns `true` if an array includes the specified element, or `false` if it does not. * This function is the mirror version of [inArray](inArray.md). * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`.

Returns a new array with an element inserted at the specified position. * The element type must match the array type. * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`.

Returns a single string by joining the elements of an array using the specified delimiter. * The element type must be compatible with string conversion. * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`.

Returns the number of elements in an array.

Returns a new array with the specified element removed. * The element type must match the array type. * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`.

Returns a new array with the element at the specified position removed. * The element type must match the array type. * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`. * Positions are **0-indexed**.

Returns a new array where all instances of a specified value are replaced with a new value. * The element type must match the array type. * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`.

Returns a new array with the element at the specified position replaced by a new value. * The element type must match the array type. * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`.

Goal

Returns a new array with the elements sorted according to the specified options. * The element type must match the array type. * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`. * By default, the array is sorted in ascending order, with null values appearing last.

Returns an array of substrings by splitting a string using the specified delimiter. * The delimiter can be either a `string` or a `regexp`.

TL;DR

Returns the average (mean) value of a numerical expression. !!! note The input must be a number. Use a cast if the field is stored as a string.

Run long-running DataPrime or Lucene queries asynchronously, and optionally save results to a system dataset for reuse across the platform.

Run long-running DataPrime or Lucene queries asynchronously and save results as Temporary for 30 days or Persistent to a user-defined summary dataset for long-term reuse.

The `between` keyword specifies a date range on which a query on `logs` or `spans` should operate. !!! note The `between` keyword will work on any time expression, but the result of the expression must be of type `timestamp`.

The `block` command filters out all documents where the given predicate evaluates to true. It is the inverse of `filter`, which keeps only documents that match the predicate.

The `bottom` command limits the rows returned from a query to the last *N* rows in a given set, ordered by a specified expression. It is useful for finding the lowest-ranking values or least frequent occurrences within a dataset. !!! note When using this command, pay close attention to your ordering expression, as it determines which records are considered "bottom" in the result.

Problem / Use case

Returns the number of bytes required to represent a UTF-8 encoded string.

Problem / Use case

Returns the number of unique elements in an array. * The element type must match the array type. * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`.

Returns a value from the first clause whose condition evaluates to `true`. Each clause is a `condition -> value` pair: when `condition` evaluates to `true`, `case` returns `value`. You can include any number of clauses, plus an optional `_ -> default` fallback. If no condition matches and no fallback is present, `case` returns `null`. !!! note `case` checks clauses top-to-bottom and returns the **first match**, so order matters.

Returns a value based on whether a string contains one of several specified substrings. This function is a shorthand for `case` expressions with [`contains()`](../string/contains.md) logic and helps shorten queries that would otherwise repeat conditional statements. If no clause matches and no `_` fallback is present, `case_contains` returns `null`. !!! warning "Behavior change" Earlier implementations of `case_contains` evaluated each clause with the [text match](../../../user-guide/foundations/understanding-expressions.md) (`~`) operator instead of `contains()`, which did not match the documented behavior. `case_contains` now correctly evaluates each clause with [`contains()`](../string/contains.md) for case-sensitive substring matching. If your query relied on the previous text-match behavior, use [`case_find`](case_find.md) instead. !!! note `case_contains` checks clauses top-to-bottom and returns the **first match**, so order matters.

Returns a value based on whether an expression equals one of several specified values. This function is a shorthand for `case` expressions with equality (`==`) logic and helps shorten queries that would otherwise repeat conditional statements. If no clause matches and no `_` fallback is present, `case_equals` returns `null`. !!! note `case_equals` checks clauses top-to-bottom and returns the **first match**, so order matters.

Returns a value based on whether a string matches one of several specified text patterns. This function is a shorthand for `case` expressions with [text match](../../../user-guide/foundations/understanding-expressions.md) (`~`) logic and helps shorten queries that would otherwise repeat conditional statements. Use `case_find` when you want pattern matching — the same behavior the [`find`](../../commands-reference/find_text.md) command uses for free-text search. For strict substring containment, use [`case_contains`](case_contains.md) instead. If no clause matches and no `_` fallback is present, `case_find` returns `null`. !!! note `case_find` checks clauses top-to-bottom and returns the **first match**, so order matters.

Returns a value based on whether a number is greater than one of several thresholds. This function is a shorthand for `case` expressions with `>` (greater than) logic and helps shorten queries that would otherwise repeat conditional statements. If no clause matches and no `_` fallback is present, `case_greaterthan` returns `null`. !!! note `case_greaterthan` checks clauses top-to-bottom and returns the **first match**, so order matters.

Returns a value based on whether a number is less than one of several thresholds. This function is a shorthand for `case` expressions with `<` (less than) logic and helps shorten queries that would otherwise repeat conditional statements. If no clause matches and no `_` fallback is present, `case_lessthan` returns `null`. !!! note `case_lessthan` checks clauses top-to-bottom and returns the **first match**, so order matters.

Returns the smallest integer greater than or equal to a number. For example, `1.5` becomes `2`, and `8.1` becomes `9`.

The `choose` command removes all keypaths **not explicitly specified**. This allows you to extract and reshape only the data you need from a larger log document. !!! note The `choose` command supports nested key paths and aliasing in the output, making it useful for simplifying complex documents.

Converts a numeric Unicode code point into its corresponding string character.

Converts a Unicode character into its numeric code point representation.

Returns an array of values collected from an expression for each group. * Supports optional deduplication (`distinct`), null filtering (`ignoreNulls`), and element limits (`limit`). * Values are aggregated into an array, preserving order of processing unless constrained by `distinct` or `limit`.

The commands available in the DataPrime Query Language

Joins multiple strings together and return the result as a single string.

Goal

Problem / Use case

Returns `true` if the given substring appears anywhere in a string; otherwise return `false`. The check is case sensitive. For case-insensitive matching, normalize both values with `toLowerCase()` or `toUpperCase()` before calling `contains`.

The `convert` command is a **semantic keyword** that indicates type conversion is taking place. It has no effect on query execution and exists purely to make transformations more readable and self-documenting. !!! note The `convert` keyword is optional and does not change functionality. Use it when you want to make type changes explicit in your query.

Problem / Use case

The `count` command returns a single row representing the total number of documents in the current result set. It can optionally store this result in a named keypath for readability using the `into` keyword.

Problem / Use case

Problem / Use case

TL;DR

TL;DR

Returns the number of rows that satisfy a given condition, counting only non-null expression values. * Useful for measuring subsets of data within groups. * Can evaluate a condition alone or in combination with a non-null expression. !!! note `count_if` is an aggregation function and must be used with a grouping keyword such as `groupby`.

The `countby` command generates a count for each distinct value in a given expression, effectively grouping the results by that key. !!! note Unlike `count`, which tallies all records in a set, `countby` provides a per-group count based on the specified key or expression.

The `create` command defines a new key and assigns it a value derived from an expression. It is one of the most flexible commands in DataPrime, allowing you to **add new fields**, **populate missing data**, or **enrich existing structures** without overwriting parent keys. In practice, `create` acts like a safe write operation for structured data. You can explicitly control what happens when a key already exists, when it is missing, or when the new value's type differs from the existing one. - **Existing keys** can be overwritten, skipped, or cause the query to fail. - **Missing keys** can be created, skipped, or trigger a failure. - **Type changes** can either be enforced, ignored, or handled as an error. !!! note Key creation is granular, meaning that parent keys in the path are not overwritten. This allows you to safely add fields within existing objects, even in nested log structures.

Problem / use case

Goal

The dataplan.quota_events dataset is a stream of quota-related events emitted by the platform — quota-definition changes, threshold breaches, ingestion blocks, Pay-as-you-go activations, and resets. Query it to track how your team's quota is allocated and consumed over time, and to investigate when and why limits are approached, exceeded, or reset.

The dataplan.usage_events dataset stores aggregated data usage events for your team. Each event captures a unit of ingestion after ratios have been applied, letting you query your team's consumption alongside any other data in Coralogix using DataPrime.

DataPrime Expression Language, or DPXL, is an expression language based on DataPrime expression syntax. Leverage it across the Coralogix platform to define rich expression-based filters.

Understanding what DataPrime is and where it fits into the Coralogix platform helps you make the most of its powerful querying capabilities. Whether you're exploring logs, building dashboards, or writing advanced queries, getting oriented with DataPrime is the first step toward unlocking unified observability across your telemetry data.

Welcome to the DataPrime User Guide. This guide helps you learn how to query and transform observability data using DataPrime, the Coralogix query language.

Return the name of the **dataset** that a record originated from, such as `logs` or `spans`. This is especially useful when a query combines records from more than one dataset — for example with [`union`](../../commands-reference/union.mdx) or [`join`](../../commands-reference/join.mdx) — and you need to know which dataset each record came from. !!! note To learn how datasets and dataspaces work, see [`source`](../../commands-reference/sources/source.mdx).

Dataset Management is the central place to view, configure, and create datasets across your dataspaces. Navigate to Data Flow, then Dataset Management.

Return the name of the **dataspace** that a record originated from, such as `default`. This is especially useful when a query combines records from more than one source — for example with [`union`](../../commands-reference/union.mdx) or [`join`](../../commands-reference/join.mdx) — and you need to know which dataspace each record came from. !!! note To learn how datasets and dataspaces work, see [`source`](../../commands-reference/sources/source.mdx).

Decodes a Base64-encoded string into its original value.

The `dedupeby` command removes duplicate documents based on one or more expressions, keeping only *N* events for each unique combination of the specified fields. This is especially useful for sampling representative data from large datasets without aggregation. Conceptually, it functions like a smart filter: it doesn’t modify event content or compute summaries—it simply trims redundancy by retaining a limited number of examples per group. Use the optional `orderby` clause to control *which* events are kept within each group—for example, the most recent entries by sorting on `$m.timestamp desc`, or the slowest requests by sorting on a latency field. Without `orderby`, the choice of which events to retain per group is not deterministic. !!! note The content of each retained document remains unchanged. `dedupeby` only limits how many documents are kept for each unique grouping.

The default dataspace is where all standard observability data (logs, spans, and custom enrichments) is routed, unless configured otherwise.

Goal

Returns the duration between two timestamps as an interval. The result is not the absolute difference: * Positive if `to > from` * Negative if `to < from`

The `distinct` command returns one document per unique value (or combination of values) for the given expressions. It is particularly useful for reporting or identifying unique entities within a dataset. Functionally, it behaves like a lightweight `groupby` without any aggregation functions—simply collapsing duplicates and returning the first occurrence of each distinct value. !!! note Use `distinct` when you want a list of unique keys, not an aggregate summary.

Returns the number of distinct, non-null values for a given expression. * Each unique value is counted once per group. !!! note `distinct_count` is an aggregation function and must be used with a grouping keyword such as `groupby`. !!! warning `distinct_count` isn't supported on the **High (Frequent Search)** tier. On `frequentsearch` data, use [`approx_count_distinct`](./approx_count_distinct.mdx) instead, which returns an approximate count of distinct values.

Returns the number of distinct, non-null values that satisfy a given condition. * A document is counted only if the condition evaluates to `true`. * Duplicate values for the same expression are counted once per group. !!! note `distinct_count_if` is an aggregation function and must be used with a grouping keyword such as `groupby`.

Returns Euler’s number `e`, a mathematical constant approximately equal to `2.718281828459045`. This value is limited to 15 decimal places.

Encodes a string into its Base64 representation.

Returns `true` if a string ends with a given substring, otherwise return `false`. !!! note Unlike `contains`, which checks for a substring anywhere in the string, `endsWith` only matches the end.

Purpose

Purpose

The `enrich` command adds contextual information to logs by performing lookups against a custom enrichment table. It merges additional columns from the lookup into each log document based on a matching key. This is particularly useful for attaching static metadata (like user details, service mappings, or IP ownership) to incoming logs without modifying upstream systems. The enrichment is applied **at query time**, meaning you always work with the most recent version of the enrichment table. Each lookup table must be created and uploaded beforehand as a **Custom Enrichment**. For setup and management instructions, see [Custom Enrichment](../../../user-guides/enrichment_rules/custom_enrichment/index.md). !!! note - All values in a lookup table are stored as strings. Use conversion functions such as `toNumber()` or `toTimestamp()` if a different type is required. - If a log already contains the enriched key, `enrich` will merge or update only the matching sub-keys; unrelated fields remain unchanged.

Goal

How Coralogix classifies data by pillar and entity type — the schema guarantees you can rely on and where they appear across the platform.

The `explode` command transforms an array of *N* elements into *N* separate documents, each containing one element of the array at the specified keypath. It’s commonly used to “flatten” nested data structures for easier analysis or aggregation. When using `explode`, you can control whether to keep or remove the original document fields via the `original` modifier: - `original discard` removes all original fields, producing minimal output with just the exploded key. - `original preserve` retains the original document fields, duplicating them across the new documents. !!! note If the destination keypath already exists in the document, it is overwritten by the exploded value. The default behavior is `original discard`.

The `extract` function allows you to transform raw strings into structured data by parsing out embedded values and storing them as objects. It supports various extraction strategies to convert unstructured fields into clean, queryable formats.

Returns a specific unit of time extracted from a timestamp, such as the hour, minute, or second. !!! note * Date units such as `'month'` or `'week'` start from **1**, not 0. * Units smaller than `minute` return floating-point numbers; all others return integers.

The `filter` command removes all documents that do not satisfy a specified condition. Only events for which the condition evaluates to `true` are retained in the result set. This command forms the foundation of most queries—it defines which data should be kept for further transformation or aggregation. Filters can be simple comparisons or complex logical expressions involving multiple conditions and functions. !!! note When comparing keypaths to `null`, the comparison only works on scalar values (`string`, `number`, `timestamp`, etc.). For nested JSON objects, comparisons with `null` will always return `null`.

The `find` command performs a free-text search within a specified keypath. It acts as a shorthand for combining `filter` with a text match (`~`). This command is ideal for quick searches across log messages or string fields where full parsing is unnecessary. The alias `text` can be used interchangeably with `find`.

TL;DR

Problem / Use case

Return the first non-null value from a list of arguments, in the order they are provided. !!! note Works only on scalar values such as `number`, `string`, or `timestamp`. Does not work on objects.

Returns the largest integer less than or equal to a number. For example, `1.5` becomes `1`, and `8.1` becomes `8`.

Returns an interval rendered as a string, with optional control over which time unit is displayed.

Returns a timestamp formatted as a string, with optional control over the output format and time zone.

Converts a string representation of a number in a given base into its numeric value. For example, `"101"` in base `2` becomes `5` in base `10`.

Returns a parsed timestamp from a numeric Unix time value. The Unix epoch starts on January 1, 1970. Timestamps before this are represented by negative numbers.

All functions available in the DataPrime Query Language

Problem / Use case

Configure a Google Cloud Storage (GCS) bucket as your Coralogix archive destination for the US3 (us-central1) environment.

TL;DR

The `groupby` command aggregates documents that share one or more common values or calculated expressions, allowing you to compute metrics such as `sum`, `avg`, `max`, `min`, and `count`. It is the cornerstone of DataPrime’s analytical capabilities, enabling powerful summarization and insight generation from raw event data. Each unique combination of grouping expressions produces a single output document. The `aggregate` or `agg` keyword specifies which aggregation functions to apply within each group. !!! note You can group by both keypaths and calculated expressions. When grouping by an expression, DataPrime evaluates it dynamically for each document before grouping.

Overview

When data enters Coralogix, it goes through a structured lifecycle: received from shippers or agents, transformed with DataPrime rules, routed based on attributes like region or team, and directed into the appropriate dataspace and dataset. If a dataset doesn't already exist, it's created automatically and inherits configuration from the parent dataspace.

Overview

Return one value if a condition is `true`, otherwise return an alternative value.

Description

Return `true` if a given value matches any value in a list of candidates, otherwise return `false`.

Returns `true` if the specified element exists within the array, or `false` if it does not. * This function is the inverse of [`arrayContains`](arraycontains.md). * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`.

Returns the position of the first occurrence of a substring within a string. The index is zero-based. If the substring is not found, the function returns `-1`.

The DataPrime Cookbook is a collection of concise, copy-pasteable recipes designed to help you solve common log analysis and observability tasks faster. Each recipe focuses on a ready-to-use query designed to answer a specific question, flag a condition, or extract a useful signal from your data, and wraps it in a compact format that’s easy to adapt.

Return `true` if an IP address belongs to a given subnet, otherwise return `false`.

Return the CIDR subnet for a given IP address and subnet size.

Returns `true` if the array contains no elements, or `false` otherwise. * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`.

Goal

Returns `true` if `array1` is a subset of `array2`, or `false` otherwise. * When comparing `array1` and `array2`, duplicates are discarded. This means two arrays of different lengths but with the same unique elements are considered equal. * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`.

Returns `true` if `array1` is a superset of `array2`, or `false` otherwise. * When comparing `array1` and `array2`, duplicates are discarded. This means two arrays of different lengths but with the same unique elements are considered equal. * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`.

Returns `true` if a given string is a valid UUID, otherwise returns `false`. Use `isUuid` to clean data or flag malformed identifiers in logs and datasets.

The `join` command merges the current (left) query with the results of a second (right) query. Conceptually, it forms a Cartesian product of left and right rows, then applies join logic based on a condition (`on`) or matching keypaths (`using`) and writes the matching right-side row into a destination keypath (`into`). **Join types** * **left** (default): Keep all left rows; attach matching right rows or `null` when no match. * **inner**: Keep only rows that match on both sides. * **full**: Keep all rows from both sides; non-matching fields are `null`. * **cross**: Return the full Cartesian product; no `on`/`using` supported. **Addressing fields** * Use `left=>` and `right=>` prefixes inside `on` to disambiguate fields with the same name; omit when a keypath exists only on one side. **Behavior & caveats** * Conditions support **equality (`==`)** on keypaths; chain multiple with `&&`. * One side must be relatively small (< 200MB); reduce size with `filter` or `remove`. * In left joins, nulls in the join keys prevent matches; use `join full` to include unmatched keys from either side. * Joins can duplicate rows if multiple matches exist; consider preprocessing (e.g., `distinct`) to avoid unintended multiplicity.

Goal

Purpose

The `last` keyword specifies how far back in time a query should go, defined by a given `interval`. !!! note The value of the given `interval` MUST be non-negative.

Returns the number of characters in a string.

The `limit` command restricts the number of documents returned by a query to a specified count. It is typically used after sorting or aggregation operations to retrieve only the most relevant or top results from a larger dataset. This command is especially useful for performance optimization or for displaying only a subset of high-value results (for example, the top 100 users or the most recent 50 logs). !!! note The `limit` command does not guarantee order unless used after an explicit `orderby`. If order matters, always pair it with `orderby` to ensure predictable results.

Limitations of the DataPrime language

Limitations of the DataPrime language

Computes the natural logarithm (base *e*) of a number.

Returns the logarithm of a number to a specified base. Common uses include modeling compound interest, exponential growth or decay, pH levels, and earthquake magnitudes.

Returns the base-2 logarithm of a number. Equivalent to `log(2, number)`.

Removes whitespace from the start of a string while leaving the end unchanged.

The `lucene` command executes a Lucene query within a DataPrime query, allowing users to seamlessly combine Lucene’s search syntax with DataPrime’s structured query capabilities. This enables powerful hybrid queries—for example, filtering or aggregating over results first narrowed by a Lucene search expression. !!! note Field names inside the Lucene query are relative to `$d` (the root level of user data). You can combine Lucene search with other DataPrime commands such as `filter`, `aggregate`, or `groupby`.

Goal

Having an excessive number of fields in your index can result in mapping issues. By default, Coralogix enforces a maximum limit of 1,000 fields to prevent performance degradation. This threshold is in place to maintain system stability and ensure optimal query performance.

Returns `true` if a string matches a given regular expression. The expression is applied to the entire string; partial matches return `false`.

Returns the largest numerical value from the input. * Can be used in aggregation to compute the maximum value across grouped rows. * When used with multiple arguments, returns the largest among them.

Returns the value of an expression associated with the maximum value of a given sort key. * Both `sortKey` and `expression` must be comparable types (`string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, or `enum`). * Useful for retrieving details from the row that has the maximum value of another field.

TL;DR

Coralogix provides a Metrics API that lets you query your hosted metrics easily.

Returns the smallest numerical value from the input. * Can be used in aggregation to compute the minimum value across grouped rows. * When used with multiple arguments, returns the smallest among them.

Returns the value of an expression associated with the minimum value of a given sort key. * Both `sortKey` and `expression` must be comparable types (`string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, or `enum`). * Useful for retrieving details from the row that has the earliest or smallest value of another field.

Returns the remainder after dividing a number by a divisor. Equivalent to the modulus operator (`%`).

Problem / Use case

The `move` command relocates a keypath to a new position within a document. This is useful when keypaths are deeply nested or inconsistently structured, making queries cumbersome to write and read. When a keypath is moved, its value (and any child keys if it’s an object) is transferred to the target keypath, and the original keypath is removed. !!! note If the source keypath is an object, the entire key and all child elements are moved to the new location.

The `multigroupby` command concatenates the results from two or more [`groupby`](groupby.md) queries into a single dataset. It allows multiple aggregation queries to execute in one scan, improving efficiency and keeping grouped results synchronized. **Key benefits:** - **Efficiency:** Data is scanned only once across multiple groupings. - **Synchronization:** Results remain coherent, avoiding mismatches between independently run groupby queries. !!! note The maximum number of buckets that `multigroupby` can process is 64.

Returns an interval multiplied by a numeric factor, allowing extrapolation over time. !!! note Both integer and decimal factors are supported.

Goal

Purpose

notification.requests is currently defined as an entity-type schema and may not yet be available as a queryable system dataset in all environments. The source system/notification.requests examples below may not return data until the dataset is registered and ingested in your environment.

Returns the current time as a timestamp at query execution. * Produces nanosecond precision if supported by the runtime, otherwise falls back to milliseconds. * Always returns the same value across multiple invocations within the same query.

The `orderby` command sorts query results in ascending or descending order based on one or more expressions. It supports multiple sort keys, allowing you to order by several fields sequentially. The command has several aliases, `orderby`, `sortby`, `order by`, and `sort by`, that behave identically. !!! note Sorting is limited to 10,000 values. Beyond that limit, order is not guaranteed.

The full glossary of the DataPrime Query Language

Welcome to the Foundations section of the DataPrime User Guide. This section covers the building blocks of DataPrime, enabling you to understand how data is structured and how the DataPrime query language works. It sets the stage for using DataPrime effectively by grounding you in its core syntax, commands, and concepts. By the end of this section, you’ll have the knowledge you need to start composing queries with confidence.

Now that you’ve learned the core building blocks of the DataPrime language— how data is structured, how commands and functions work, and how to run a query— you’re ready to start solving real problems.

Dataspaces and datasets provide a two-tiered model for organizing, routing, and securing observability data in Coralogix.

Schema management offers a unified approach to governing your logs—letting you both discover the structure of ingested data and enforce the fields that matter most.

The system dataspace provides powerful visibility into the structure, behavior, and configuration of your organization's data. You can track how schemas evolve, review alert activity, and inspect audit events — all of which support debugging, auditing, and operational insight.

**Alias for [`padLeft`](./padleft.md).** Adds characters to the beginning of a string until it reaches the desired length. If the string is longer than the target length, it is truncated from the end. !!! note The `fillWith` argument must be a single-character string.

Adds characters to the beginning of a string until it reaches the desired length. If the string is longer than the target length, it is truncated from the end. !!! note The `fillWith` argument must be a single-character string.

Adds characters to the end of a string until it reaches the desired length. If the string is longer than the target length, it is truncated from the end. !!! note The `fillWith` argument must be a single-character string.

Returns an interval parsed from a string representation such as `2d` or `35m10s`, enabling calculations with durations. A valid string must follow these rules: * Format: `NdNhNmNsNmsNusNns` where `N` is a non-negative integer * At least one time unit must be present * No unit may appear more than once (`1d2d` is invalid) * Units must appear in descending order (days → nanoseconds). `1d1s` is valid; `1s1d` is not * A leading `-` is allowed to indicate a negative interval (the only valid position for `-`) * If the format is invalid, the function returns `null`

Returns a parsed timestamp from a date or time string, enabling use of DataPrime's time functions. !!! note If the string cannot be parsed (for example, if it does not match the expected format), the function returns `null`.

Returns a timestamp parsed from a date/time string with an optional format specification and time zone override. See [`parseTimestamp`](./parsetimestamp.md) for details.

Problem / Use case

Returns the approximate n-th percentile of a numerical expression. * The percentile value must be between `0` and `1`. * The calculation is approximate, with accuracy controlled by the optional `error_threshold`.

Returns the mathematical constant π (pi), limited to 15 decimal places.

Returns the result of raising a number to the power of an exponent.

This guide explains why quoting can break your HTTP API requests and how to structure payloads safely and cleanly.

Returns a pseudorandom decimal between `0` (inclusive) and `1` (exclusive). !!! note `random` is not cryptographically secure.

Returns a pseudorandom integer between `0` (inclusive) and an upper bound (exclusive). !!! note `randomInt` is not cryptographically secure.

Returns a randomly generated UUIDv4, useful for assigning identifiers to documents.

Return the cloud storage location of a record, such as the URL of an S3 object when using AWS S3. !!! note Works only with data stored in your cloud storage. Running this on indexed documents (Frequent Search mode) will return `null`. Use it alongside [`dataset()`](./dataset.mdx) and [`dataspace()`](./dataspace.mdx) to capture the full source of a record — its dataspace, dataset, and exact storage location.

The `redact` command replaces parts of a string that match a given substring or regular expression with a replacement value. It’s commonly used to hide sensitive information such as emails, tokens, or identifiers found in message fields. You can use either a plain string or a regular expression pattern to define what should be redacted. !!! note The optional keyword `matching` improves readability but is not required.

Splits a string using a regular expression as the delimiter and return the token at the specified index. The index starts at 1, not 0.

Relational queries in Coralogix's Query Builder empower you to analyze and understand the intricate relationships within your distributed traces.

The `remove` command deletes one or more keypaths from every document in the working set. It is the inverse of `choose`, which keeps only the specified fields. This is especially useful for sanitizing logs or removing fields containing unnecessary or sensitive information before performing analysis. !!! note `remove` can operate on scalar values or entire objects.

The `replace` command overwrites the value of an existing keypath with the result of a new expression. It is often used to clean or transform existing values while maintaining the same document structure. This is particularly helpful for decoding, normalizing, or updating data fields without creating new keys.

Explicitly define fields of importance for querying and monitoring purposes.

Returns a number rounded to the nearest integer or to a specified number of decimal places. For example, `1.5` becomes `2` and `8.1` becomes `8`.

Returns an interval rounded down to a specified precision. All time units smaller than the specified `timeunit` are zeroed out.

Returns a timestamp rounded down to the nearest interval. !!! note Functionally equivalent to dividing a timestamp by an interval: `timestamp / interval`.

Removes whitespace from the end of a string while leaving the beginning unchanged.

Returns the sample standard deviation of a numerical expression within a group. * Designed for use cases where substantial values are missing, such as when data is heavily sampled. !!! note Use `sample_stddev` when working with incomplete datasets (e.g., sampled trace data). For full datasets, prefer `stddev`.

Returns the sample variance of a numerical expression within a group. * Useful for analyzing variability when working with incomplete datasets. * Designed for scenarios like heavy trace sampling, where global variance would be misleading. !!! note Use `sample_variance` when only a subset of data is available. For complete datasets, prefer `variance`.

Explicitly define fields of importance for querying and monitoring purposes.

Returns the set difference of two arrays, producing a new array with elements from `array1` that are not in `array2`. * Duplicates are discarded when computing the difference. * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`.

Returns the symmetric difference of two arrays, producing a new array with elements that exist in either `array1` or `array2` but not in both. * Duplicates are discarded when computing the difference. * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`.

Returns `true` if `array1` and `array2` contain the same unique elements, or `false` otherwise. * When comparing arrays, duplicates are discarded. This means two arrays of different lengths but with the same unique elements are considered equal. * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`.

Returns the intersection of two arrays, producing a new array with elements common to both. * Both arrays are treated as sets: * Duplicates are removed from both arrays * Order is not preserved in the result * `null` is treated as an empty set * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`.

Returns the union of two arrays, producing a new array that contains all unique elements from both. * Both arrays are treated as sets: * Duplicates are removed * Order is not preserved in the result * `null` is treated as an empty set * Supported element types include `string`, `bool`, `number`, `interval`, `timestamp`, `regexp`, and `enum`.

The `source` command is a foundational component of DataPrime. It informs the DataPrime engine which datasource you wish to read from. !!! note While you can start your query with this, the `source` command is optional and will default to `logs`. ### Basic usage In DataPrime, you read data by specifying a **dataset** within an optional **dataspace**: ```dataprime source <dataspace>/<dataset> ``` If no dataspace is provided, the query defaults to the `default` dataspace. This allows for concise syntax when working within the most common data sources. Common datasets include: * `logs` – Application and infrastructure logs. *Default dataset. Equivalent to `source default/logs`.* * `spans` – Distributed tracing data from systems like OpenTelemetry. *Equivalent to `source default/spans`.* * `enrichments/<name>` – [custom enrichment](../../../../user-guides/enrichment_rules/custom_enrichment/index.md) tables uploaded via the UI or API. *For example: `source default/enrichments/ip_lookup`* You can also query your **High (Frequent Search)** tier directly through the `frequentsearch` dataspace: * `frequentsearch/logs` – High-priority logs kept in the Frequent Search (hot) tier. * `frequentsearch/spans` – High-priority spans kept in the Frequent Search (hot) tier. Because `frequentsearch` datasets behave like any other source, you can reference them in [`join`](../join.mdx) and [`union`](../union.mdx) operations alongside `default` and `system` data. For more, see the [data layer overview](../../../../user-guides/data-layer/overview.mdx). You can also query system-generated datasets such as: * `system/engine.queries` – Logs of all DataPrime query executions. * `system/alerts.history` – Historical records of alert events. > Dataset names may include dots (e.g., `engine.queries`) but are still treated as flat identifiers—not nested structures. This structure supports querying across teams, environments, or pipelines—whether you’re debugging logs, analyzing performance, or auditing notifications.

The DataPrime functions parseTimestamp and formatTimestamp accept a format argument that can be used to specify the format for parsing, respectively printing a timestamp to a string. The syntax is based on the strftime function from programming languages such as Python, C or Rust. It can be any valid string with embedded format specifiers as detailed in the table below. Other parts of the string which are not format specifiers are reproduced verbatim.

Splits a string using a delimiter and return the token at the specified index. The index starts at 1, not 0.

Returns the square root of a number. !!! note This is the inverse of `power(number, 2)`.

Returns `true` if a string begins with a given substring, otherwise return `false`. !!! note Unlike `contains`, which checks for a substring anywhere in the string, `startsWith` only matches the beginning.

Returns the standard deviation of a numerical expression within a group. * Useful for measuring how much values vary around the mean. * Best applied when a complete dataset is available (for sampled data, use `sample_stddev`).

The `stitch` command performs a **horizontal union** of two datasets, combining them side-by-side. It aligns rows from one dataset with rows from another and merges their columns into a single unified dataset. This is particularly useful for joining datasets that share a logical order but lack a join key. **Key behaviors:** - Rows are combined in order (row 1 with row 1, row 2 with row 2, etc.). - If one dataset contains more rows than the other, unmatched rows include `null` values for missing columns. - The resulting dataset contains all columns from both sides. **Difference from `union`:** - `stitch` merges datasets horizontally (adding columns). - `union` merges datasets vertically (adding rows).

Extracts a substring from a string starting at a given position, with an optional length. Useful for simple value extraction without needing regular expressions.

Returns the result of subtracting one interval from another. !!! note Equivalent to `addInterval(left, -right)` or `left - right`.

Returns a timestamp reduced by a given interval. !!! note Equivalent to `addTime(t, -i)` or `t - i`, where `t` is a timestamp and `i` is an interval.

Returns the total of all numerical values passed into the function. * Can be used in aggregation to compute totals across grouped rows. * When used with multiple arguments, returns the sum of all provided values.

Problem / Use case

Searches for a text phrase within a target value. The behavior depends on the type of the target: * **Primitive types** (`string`, `number`, `bool`, `interval`, `timestamp`, `regexp`, `enum`) are converted to strings before searching. * **Objects** are searched by their values (keys are ignored). * **Arrays** are searched by their elements. The `phrase` must appear as one or more complete tokens. Tokens are defined during log parsing and are split on the following characters: whitespace (`\s`), `=`, `/`, `:`, `@`, `#`, `$`, `*`, `|`, `,`, `;`, `'`, `"`, `(`, `[`, `{`, `}`, `)`, `]`, `&lt;`, `&gt;`, `.`, `_`, `-`. Because of this, a search for `online` will not match `onlineboutique` (one token), but will match `online_boutique` (`online` and `boutique` are separate tokens). Similarly, `Version 17` matches `Version 17.4 (Build 21E213)` because `Version` and `17` are two tokens, but `Vers` would not match because it is not a full token.

Goal

Returns a timestamp rounded down to a specified interval. This function is deprecated in favor of `roundTime`.

The `timeshifted` keyword modifies the timerange within the scope of the query. This is useful when, for example, a custom dashboard has a global timerange of `Today` but your query only makes sense when looking at yesterday's data.

Converts a number into its string representation in a specified base. For example, converting `10` into base `16` results in `"a"`.

Returns an interval created from a numeric value and an optional time unit. This function works with integers, decimals, positive, and negative values.

Returns a timestamp formatted as an ISO 8601 string (e.g. `2023-08-11T07:29:17.634Z`). This function supports nanosecond precision.

Converts all alphabetical characters in a string to lowercase.

The `top` command returns the first N results after sorting by one or more expressions in the `by` clause. It can be used with plain expressions or with aggregation functions. !!! note - Without aggregation, `top` limits the full result set to N rows, ordered by a given expression. - With aggregation, `top` groups results by the result expressions and returns the top N groups (not N rows per group), ranked by the ordering expression. - Sorting direction (ascending/descending) is determined by the expression or implicit aggregate ordering. !!! note `top` with aggregation returns top N **groups overall**, not N rows per group.

Returns an interval converted to a numeric value in the requested time unit. Use `toTimeUnit` when you need an interval as a plain number — for example, to report a duration in seconds, plot it on a numeric chart, or compare it against a numeric threshold.

Returns the number of time units since the Unix epoch (`1970-01-01T00:00:00Z`) for a given timestamp. !!! note Timestamps before the epoch are represented as negative numbers.

Converts all alphabetical characters in a string to uppercase.

Removes whitespace from both the start and end of a string.

Common issues and fixes

| Type | Description | Encoding |

Goal

Goal

Goal

Goal

Goal

Goal

The `union` command concatenates the results from two or more datasets into one dataset. This allows users to combine results from multiple queries into one seamless dataset. One dataset can be a result set piped into the `union` command and then concatenated with another dataset.

Returns the decoded version of a URL-encoded string. URL encoding replaces certain characters with escape sequences (for example, spaces become `%20`) so they can be safely transmitted in URLs. Use `urlDecode` to restore the original values after parsing.

Returns the URL-encoded version of a string. URL encoding replaces special characters with escape sequences (for example, spaces become `%20`) so values can be safely transmitted in URLs. Use `urlEncode` when preparing data for output, such as with log forwarders.

Create custom datasets under the default dataspace to isolate log streams, apply access policies, and route data via TCO Optimizer.

Returns a randomly generated UUID, useful for assigning identifiers to documents.

Returns the variance of a numerical expression within a group. * Variance measures how far values spread out from the mean. * Best applied when a complete dataset is available (for sampled data, use `sample_variance`).

DataPrime is Coralogix's piped syntax language, offering users a straightforward yet powerful tool for describing event transformations and aggregations.

Welcome to the DataPrime Advanced Guide.

The `wildfind` command searches for a given string across **all keypaths** in every document in the working set. It is useful when you don’t know which field contains the target value. The alias `wildtext` behaves identically. !!! note `wildfind` is significantly slower than `find` or `text`, since it must inspect every field in every document. Use `find` when the keypath is known for better performance.