Skip to content

Expressions

Language constructs

This section provides an overview of the language constructs supported by DataPrime. These constructs serve as the building blocks for complex query expressions, combining operations to communicate what they want to achieve or calculate.

The constructs are:

  • Validated during query writing.
  • Executed at runtime.
  • Runtime failures are returned as null.
  • In boolean predicates, null values are treated as false.

These constructs are effectively expressions as well, and are the building blocks which allow users to “compose” more complex expressions. Let’s discuss this separately.

Literals

Literals are constant values that can represent different data types. They are the most basic form of data and are used to express fixed values directly in the query.

Examples:

  • Boolean: true, false
  • String: "example", 'single-quoted string'
  • Number: 123, 45.67
  • Array: [1, 2, 3], ["one", "two", "three"]
  • Null: null

Example literals for all data types can be found here.

Nested field access

Fields inside objects can be accessed using dot-notation or map-notation. This is useful when working with nested data.

Examples:

  • Dot Notation: user.name (to access the name field inside user)
  • Map Notation: $d['my-special.log-key'] (for fields with special characters in their keys)

Math operations

Basic mathematical operations allow you to perform arithmetic calculations.

Supported operations:

Operation Symbol
Addition +
Subtraction -
Multiplication *
Division /
Modulus %

Example:

5 + 10 * 2  // Result: 25

Boolean operations

Boolean operations are used to combine or negate boolean expressions.

Supported operations:

Operation Symbol
AND &&
OR ```
NOT !

Example:

true && false  // Result: false

Comparison operations

Comparison operations allow you to compare two values and return a boolean result.

Supported operations:

Operation Symbol
Equals ==
Not Equals !=
Greater Than >
Less Than <
Greater Than or Equals >=
Less Than or Equals <=

Examples:

x > 10 && x <= 20  // Check if x is larger than 10 and equal to or less than 20
source logs | filter response_time > 500 && status_code == 200 

//Filters logs where the response time is greater than 500 milliseconds and the status code is 200

Text search operations are used to match patterns within strings using regular expressions.

Supported Constructs:

  • Regex Match: ~ (matches a pattern)
  • Fuzzy Match: ~~ (wildfind which works on $d)

Example:

name ~ /[A-Za-z]+/  // Matches names with letters only

String concatenation

String concatenation is the joining of two or more string fields, such that they produce a single string that is made up of each original string. Concatenation is done using the concat function. For example, consider the following document:

{
    "stringa": "Hello ",
    "stringb": "World!"
}

We can concatenate these two values, like so:

create stringc from stringa.concat(stringb)

Resulting in the following documents:

{
    "stringa": "Hello ",
    "stringb": "World!",
    "stringc": "Hello World!"
}

String interpolation

String interpolation allows dynamic value concatenation within strings, improving readability.

Syntax:

  • A string is wrapped in backticks (```).
  • Inside the string, expressions are enclosed in {}.

Examples:

`The value of x is {x}`  // Inserts the value of x into the string

`Alert {alert_name} has been triggered at {timestamp}`

`This user has an ID of: {user_id}`

If you want to generate a custom message that includes a user ID from your logs, you can do the following:

source logs | filter user_id != null | replace custom_message with `User ID: {user_id} logged in.`

This query filters logs to include only those with a user ID, then creates a custom message incorporating that user ID.

Time-based operations

Time constructs allow you to work with timestamps and intervals. You can represent, manipulate, and compare time-related data.

Supported formats:

  • Timestamp Literal: @<timestamp>, @<iso8601time>, @(<expression>)
  • Interval Literal: 1h, 30m, 2d

Examples:

  • Time Bucket: timestamp / 1h (to group timestamps by hour)
  • Time Arithmetic: @2021-01-01T12:00:00 - 1h (to subtract one hour from a timestamp)

See Working with time in DataPrime for a full review of time-based operations.

Data type casting

Casting allows you to convert one data type into another. This is useful when you need to manipulate or transform data into a different form.

Syntax:

:<new-type>

Example:

123 :string  // Converts the number 123 into a string "123"

Comments

Comments can be added to queries to explain the logic or temporarily disable code.

Supported comment styles:

  • Single-line comment: //
  • Multi-line comment: /* ... */

Example:

// This is a single-line comment
/*
   This is a multi-line comment
*/

In DataPrime, there are two types of text searches: free-text search and wild-text search.

  • Free-text search refers to searching within a specific key or field in the event data.
  • Wild-text search refers to searching across the entire event data, not restricted to a particular field.

Free-text search

  • Syntax: <keypath> ~ <some-string>
  • This type of search allows you to filter by a specific key, similar to Lucene's key:text construct.
  • It can be used as part of filter conditions in queries to match a string within a specific field.

Wild-text search

  • Syntax: $d ~~ <some-string>
  • This search is applied across all fields of the event data, not just one specific field. It behaves similarly to Lucene’s <text> construct, allowing for broader searches.
  • Like free-text, it can also be used within filters.

Lucene operator

  • DataPrime supports a lucene operator that can take a full Lucene query as a string, enabling complex text searches using Lucene’s syntax directly.

Query optimization

  • It's recommended to place text searches early in the query to optimize performance.
  • The system has been optimized to the point where, in some cases, wild-text search may be faster than free-text search for broader queries.