enrich - Enrich logs with contextual information

The enrich command will add additional fields into a log, by utilizing a lookup table. You can upload your lookup table by adding a new custom enrichment lookup table. For more details, see Custom Enrichment

Note

All values in a lookup table are considered as strings. If you wish to interpret them as more appropriate datatypes, this can be done with the relevant function.

Syntax

enrich <value_to_lookup> into <enriched_key> using <lookup_table>

value_to_lookup - A string expression that will be looked up in the lookup table.
enriched_key - Destination key to store the enrichment result in.
lookup_table - The name of the Custom Enrichment table to be used.

The table's columns will be added as sub-keys to the destination key. If value_to_lookup is not found, the destination key will be null.
You can then filter the results using the DataPrime capabilities, such as filtering logs by a specific value in the enriched field.

Reading an existing lookup table

The DataPrime source keyword can be used to view the contents of a lookup table, like so:

source <name of table>

For example, if our lookup table is called my_users then source my_users will return the contents of this lookup table.

Example - Adding employee information to a User ID

Consider the following log document, with a field userid:

{
    "userid": "111",
    ...
}

And consider the following lookup table, called my_users and uploaded into the Coralogix platform:

ID	Name	Department
111	John	Finance
222	Emily	IT

We can then bring in ID, Name and Department as additional, enriched fields into our logs, using the enrich keyword:

enrich userid into user_enriched using my_users

Which produces the following enriched log:

{
    "userid": "111",
    "user_enriched": {
        "ID": "111",
        "Name": "John",
        "Department": "Finance"
    },
    ...
}

Example - Overwriting older documents with existing data.

If older events were already enriched using an older version of the lookup table, then their document will look something like this:

{
    "userid": "111",
    "user_enriched": {
        "ID": "111",
        "Name": "John",
        "Department": "Finance"
    },
    ...
},
{
    "userid": "222",
    "user_enriched": {
        "ID": "222",
        "Name": "Emily",
        "Department": "IT"
    },
    ...
}

Now, let's consider the following lookup table, Where John is no longer a member of the Finance department. He has moved to Marketing and a new version of the lookup table has been defined for this:

ID	Name	Department
111	John	Marketing
222	Emily	IT

The enrich operator can be used to overwrite older documents on read to ensure that all analytics are performed with up to date information:

enrich userid into user_enriched using my_users

Now, the result of this query is as follows:

{
    "userid": "111",
    "user_enriched": {
        "ID": "111",
        "Name": "John",
        "Department": "Marketing" # Note John's department has been updated.
    },
    ...
},
{
    "userid": "222",
    "user_enriched": {
        "ID": "222",
        "Name": "Emily",
        "Department": "IT"
    },
    ...
}

Key details:

If the original log already contains the enriched key:
- If <value_to_lookup> exists in the <lookup_table>, the sub-keys will be updated with the new value. If the <value_to_lookup> does not exist, their current value will remain.
- Any other sub-keys which are not columns in the <lookup_table> will remain with their existing values.
All values in the <lookup_table> are considered to be strings. This means that:
- The <value_to_lookup> must be in a string format.
- All values are enriched in a string format. You may then convert them to your preferred format (e.g. JSON, timestamp) using the appropriate functions.

distinct

Next explode