Expressions
Language constructs
This section provides an overview of the language constructs supported by DataPrime. These constructs serve as the building blocks for complex query expressions, combining operations to communicate what they want to achieve or calculate.
The constructs are:
- Validated during query writing.
- Executed at runtime.
- Runtime failures are returned as
null
. - In boolean predicates,
null
values are treated asfalse
.
These constructs are effectively expressions as well, and are the building blocks which allow users to “compose” more complex expressions. Let’s discuss this separately.
Literals
Literals are constant values that can represent different data types. They are the most basic form of data and are used to express fixed values directly in the query.
Examples:
- Boolean:
true
,false
- String:
"example"
,'single-quoted string'
- Number:
123
,45.67
- Array:
[1, 2, 3]
,["one", "two", "three"]
- Null:
null
Example literals for all data types can be found here.
Nested field access
Fields inside objects can be accessed using dot-notation or map-notation. This is useful when working with nested data.
Examples:
- Dot Notation:
user.name
(to access thename
field insideuser
) - Map Notation:
$d['my-special.log-key']
(for fields with special characters in their keys)
Math operations
Basic mathematical operations allow you to perform arithmetic calculations.
Supported operations:
Operation | Symbol |
---|---|
Addition | + |
Subtraction | - |
Multiplication | * |
Division | / |
Modulus | % |
Example:
Boolean operations
Boolean operations are used to combine or negate boolean expressions.
Supported operations:
Operation | Symbol |
---|---|
AND | && |
OR | ``` |
NOT | ! |
Example:
Comparison operations
Comparison operations allow you to compare two values and return a boolean result.
Supported operations:
Operation | Symbol |
---|---|
Equals | == |
Not Equals | != |
Greater Than | > |
Less Than | < |
Greater Than or Equals | >= |
Less Than or Equals | <= |
Examples:
source logs | filter response_time > 500 && status_code == 200
//Filters logs where the response time is greater than 500 milliseconds and the status code is 200
Text search
Text search operations are used to match patterns within strings using regular expressions.
Supported Constructs:
- Regex Match:
~
(matches a pattern) - Fuzzy Match:
~~
(wildfind which works on $d)
Example:
String concatenation
String concatenation is the joining of two or more string fields, such that they produce a single string that is made up of each original string. Concatenation is done using the concat
function. For example, consider the following document:
We can concatenate these two values, like so:
Resulting in the following documents:
String interpolation
String interpolation allows dynamic value concatenation within strings, improving readability.
Syntax:
- A string is wrapped in backticks (```).
- Inside the string, expressions are enclosed in
{}
.
Examples:
`The value of x is {x}` // Inserts the value of x into the string
`Alert {alert_name} has been triggered at {timestamp}`
`This user has an ID of: {user_id}`
If you want to generate a custom message that includes a user ID from your logs, you can do the following:
This query filters logs to include only those with a user ID, then creates a custom message incorporating that user ID.
Time-based operations
Time constructs allow you to work with timestamps and intervals. You can represent, manipulate, and compare time-related data.
Supported formats:
- Timestamp Literal:
@<timestamp>
,@<iso8601time>
,@(<expression>)
- Interval Literal:
1h
,30m
,2d
Examples:
- Time Bucket:
timestamp / 1h
(to group timestamps by hour) - Time Arithmetic:
@2021-01-01T12:00:00 - 1h
(to subtract one hour from a timestamp)
See Working with time in DataPrime for a full review of time-based operations.
Data type casting
Casting allows you to convert one data type into another. This is useful when you need to manipulate or transform data into a different form.
Syntax:
Example:
Comments
Comments can be added to queries to explain the logic or temporarily disable code.
Supported comment styles:
- Single-line comment:
//
- Multi-line comment:
/* ... */
Example:
Free/Wild text search
In DataPrime, there are two types of text searches: free-text search and wild-text search.
- Free-text search refers to searching within a specific key or field in the event data.
- Wild-text search refers to searching across the entire event data, not restricted to a particular field.
Free-text search
- Syntax:
<keypath> ~ <some-string>
- This type of search allows you to filter by a specific key, similar to Lucene's
key:text
construct. - It can be used as part of filter conditions in queries to match a string within a specific field.
Wild-text search
- Syntax:
$d ~~ <some-string>
- This search is applied across all fields of the event data, not just one specific field. It behaves similarly to Lucene’s
<text>
construct, allowing for broader searches. - Like free-text, it can also be used within filters.
Lucene operator
- DataPrime supports a
lucene
operator that can take a full Lucene query as a string, enabling complex text searches using Lucene’s syntax directly.
Query optimization
- It's recommended to place text searches early in the query to optimize performance.
- The system has been optimized to the point where, in some cases, wild-text search may be faster than free-text search for broader queries.