enrich - Enrich logs with contextual information
The enrich
command will add additional fields into a log, by utilizing a lookup table. You can upload your lookup table by adding a new custom enrichment lookup table. For more details, see Custom Enrichment
NOTE: All values in a lookup table are considered as strings. If you wish to interpret them as more appropriate datatypes, this can be done with the relevant function.
Syntax
-
value_to_lookup
- A string expression that will be looked up in the lookup table. -
enriched_key
- Destination key to store the enrichment result in. -
lookup_table
- The name of the Custom Enrichment table to be used.
The table's columns will be added as sub-keys to the destination key. If value_to_lookup
is not found, the destination key will be null
.
You can then filter the results using the DataPrime capabilities, such as filtering logs by a specific value in the enriched field.
Reading an existing lookup table
The DataPrime source
keyword can be used to view the contents of a lookup table, like so:
For example, if our lookup table is called my_users
then source my_users
will return the contents of this lookup table.
Example - Adding employee information to a User ID
Consider the following log document, with a field userid
:
And consider the following lookup table, called my_users
and uploaded into the Coralogix platform:
ID | Name | Department |
---|---|---|
111 | John | Finance |
222 | Emily | IT |
We can then bring in ID
, Name
and Department
as additional, enriched fields into our logs, using the enrich
keyword:
Which produces the following enriched log:
Example - Overwriting older documents with existing data.
If older events were already enriched using an older version of the lookup table, then their document will look something like this:
{
"userid": "111",
"user_enriched": {
"ID": "111",
"Name": "John",
"Department": "Finance"
},
...
},
{
"userid": "222",
"user_enriched": {
"ID": "222",
"Name": "Emily",
"Department": "IT"
},
...
}
Now, let's consider the following lookup table, Where John
is no longer a member of the Finance
department. He has moved to Marketing
and a new version of the lookup table has been defined for this:
ID | Name | Department |
---|---|---|
111 | John | Marketing |
222 | Emily | IT |
The enrich
operator can be used to overwrite older documents on read to ensure that all analytics are performed with up to date information:
Now, the result of this query is as follows:
{
"userid": "111",
"user_enriched": {
"ID": "111",
"Name": "John",
"Department": "Marketing" # Note John's department has been updated.
},
...
},
{
"userid": "222",
"user_enriched": {
"ID": "222",
"Name": "Emily",
"Department": "IT"
},
...
}
Key details:
-
If the original log already contains the enriched key:
-
If
<value_to_lookup>
exists in the<lookup_table>
, the sub-keys will be updated with the new value. If the<value_to_lookup>
does not exist, their current value will remain. -
Any other sub-keys which are not columns in the
<lookup_table>
will remain with their existing values.
-
-
All values in the
<lookup_table>
are considered to be strings. This means that:-
The
<value_to_lookup>
must be in a string format. -
All values are enriched in a string format. You may then convert them to your preferred format (e.g. JSON, timestamp) using the appropriate functions.
-