Data types
Supported data types
DataPrime supports the following data types.
Data Type | Explanation | Reflected in JSON |
---|---|---|
Timestamp | A precise point in time in nanoseconds. Often used for tracking events or logs. | Represented as a JSON number in nanoseconds. Example: 1609459200000000 (for a specific timestamp). |
String | A sequence of characters. Commonly used for textual data, identifiers, or other alphanumeric data. Internally, the engine uses utf-8 encoding for all strings. | Represented as a JSON string. Example: "example text" . |
Number | A numeric value, which can be an integer or a floating point. Used for calculations or measurements. | Represented as a JSON number. Example: 123 or 45.67 . |
Boolean | Represents a true or false value. Common in logical checks and conditions. | Represented as a JSON boolean. Example: true or false . |
Interval | A time duration (e.g., 1m , 5h , 3h30m ). Supported time units are: d , h , m , s , ms , us , ns . | Represented as a JSON string when shown as an interval. Example: "1h" . Casted to a JSON number in nanoseconds (e.g., 3600000000000 for 1h ). |
Null | Represents the absence of a value. Commonly used to indicate missing or undefined data. | Represented as JSON null . Example: null . |
Regexp | A regular expression used for pattern matching in strings. | Represented as a JSON string containing the regular expression. Example: "/[a-z]+/" . (Usually not used directly in JSON output). |
Array | A collection of values of a primitive type. | Represented as a JSON array. Example: ["a", "b", "c"] . |
For intervals, although JSON doesn't natively support them, they can be displayed as strings or cast into a number when needed (e.g., in nanoseconds).
Data type tracking
The system tracks data types on raw data and at each stage of the query, based on the results of previous operations. This means the query engine and parsers maintain a record (or lineage) of data types and any conversions that occur during the query process. This tracking enables better validation, autocompletion, and suggestion capabilities when building queries.
Data type conversions
The system supports converting data types in several ways, using core constructs, operators and functions.
Casting
DataPrime users can use casting to transform the datatype of a given keypath. For example, in the following field, the user may change the type of status_code
to a number, to perform a greater-than filter:
Using the convert
command
convert
is a command for changing the type of a field. This is most useful when you wish to make multiple downstream calculations with a field after conversion, and you don’t wish to cast every time.
The datatypes
clause
In various commands, there is a datatypes
clause that can provide type information to the DataPrime query engine, which ensures that when values are parsed or extracted, they are done so into the correct type. For example, consider the following document:
We can use the extract
command to pull out the values of these different fields, but query_results_duration_ms
is clearly a numerical value, and we should inform DataPrime that it is. Using the extract
command, the kv()
parsing strategy and the datatypes
clause, we can do everything in one command:
Conversion functions
Throughout the DataPrime Functions Reference, there are functions designed to help to parse or transform a given value. For example, interpreting a numerical value as a time interval: