Custom Data Enrichment

Overview

Custom Log Enrichment with Coralogix enables you to easily enrich your log data. Now, you can automatically add fields to your JSON logs based on specific matches in your log data, using a pre-defined custom data source of your own. This way, you can enhance your log data with business, operations, or security information that is not available at runtime.

There are two possible ways to enrich your logs:

• Select a log key to look up for a key value and enrich the logs automatically during ingestion. The logs are saved with the enriched fields. The advantages of this mode:
  • Logs are automatically enriched.
  • The logs themselves include the enrichment data, which makes it easier to consume everywhere (by any query, and also by third-party products that read the logs from the S3 bucket).
• Use the DataPrime query enrich to look up for a value in this table and enrich the log dynamically for the purpose of the query. The advantages of this mode:
  • It allows you to enrich old logs already ingested into Coralogix.
  • The enrichment does not increase the size of the stored logs, as the enrichment is done dynamically, only for the query results.
  • For more information on the DataPrime enrich query, see the DataPrime Glossary.

How Can I Use Data Enrichment?

Take a look at these use-cases to get a feel for the many ways that Data Enrichment can serve you.

Monitoring

Assuming we have a log with a UUID that represents a customer, without an actual field that holds the customer name. The goal is to have a field in the log that will contain the customers’ names so we can visualize/search our logs based on them. With custom enrichment, we can create such enrichment by setting up a CSV file to map each UUID to a customer name.

Security

Given a field with a domain name in our log that represents where our applications were accessed from. We want to create an alert that will notify us if there was any attempt to access our application from an unauthorized domain. For that purpose, we can set a CSV file with a list of the whitelisted domains so that every log with one of these domains will be enriched with a field that will contain the word ‘allowed’, and then we can create an alert for those logs that will not contain this field (e.g. the query will look like: NOT domain_enriched:allowed).

Configuration

STEP 1. Prepare Your CSV File.

To get started, first create your CSV file that will define your custom data source. There are 2 supported types of data enrichment, string to string, or string to a JSON object.

For string to string the file should include 2 columns:

• Column A will include a list of values to match
• Column B will include the enrichment information (the file must include headers although column names have no meaning).

Note: Only the string to string option is supported by the DataPrime query enrichment mode.

For string to JSON, the file should include at least 3 columns:

• Column A will include a list of values to match,
• Columns B, C, etc.. will include the enriched JSON object fields’ information (the file must include headers, which will then be the enriched JSON fields names correspondingly).

Note – file size is limited to 10,000 rows (~0.5MB of standard information).

Example of a string–>string CSV file that will contain pairs of UUID and customers’ names to allow better readability to the business and support teams.

Example of a string–>JSON CSV file that will contain in each row a UUID, customer name, and customer size.

STEP 2. Upload Your CSV File.

Once your CSV file is ready, upload it to Coralogix Custom Enrichment page:

• Navigate in the Coralogix toolbar to Data Flow –> Data Enrichment.
• Scroll down to the Custom Enrichment section and click ADD CUSTOM ENRICHMENT.

• Name your new enrichment and provide an (optional) description.
• Select and upload the CSV file you created in the previous step.
• Click CREATE.

STEP 3. Enriching Your Logs

There are two ways to use the enrichment feature:

Option 1: Configure the key to enrich automatically while the log is being ingested (see below).

Option 2: Enrich using the DataPrime query. To learn more about how to enrich your logs using the DataPrime query, see the enrich command in the DataPrime Glossary.

Configure the Key for Automatic Enrichment

In the Custom Enrichment section, you can define fields from your logs whose values should be compared against the CSV file values. When a log record contains one of these fields with its value matching any value from the CSV file (first column), a new field called field_enriched will be added to the JSON record and its value will be the corresponding value in the next column(s).

Example

Let’s see how it looks in Coralogix. These are the original logs that are being sent, using the IDs from the CSV files mentioned above.

Using string–>string enrichment:

Using string–>JSON enrichment:

API Support

1. Please follow these steps to create a Custom Enrichment using the Coralogix API:

• Make sure to select the correct API endpoint for your cluster:

Cluster	API Endpoint
.us	https://webapi.coralogix.us/api/v1/external/custom-enrichments
.in	https://api.app.coralogix.in/api/v1/external/custom-enrichments
.com	https://webapi.coralogix.com/api/v1/external/custom-enrichments
eu2.coralogix.com	https://webapi.eu2.coralogix.com/api/v1/external/custom-enrichments
coralogixsg.com	https://webapi.coralogixsg.com/api/v1/external/custom-enrichments

• Use the Alerts, Rules and Tags API Key as the Bearer token.
  (Click on the Data Flow top menu –> API Keys. [If you have not yet generated the key, please click “GENERATE NEW API KEY”]. Copy the key there).

2. The following Custom Enrichments API calls are supported:

• Creation
• Update
• Deletion
• Listing

3. Create a new Custom Enrichment (copy the curl commands below and customize them for your environment before sending them to the API. Please remember to use the correct endpoint for your deployment as described earlier in this tutorial):

Request:

curl --location --request POST 'https://webapi.coralogix.com/api/v1/external/custom-enrichments' \  
--header 'Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \ 
--form 'name="\"YOUR_ENRICHMENT_NAME\""' \ 
--form 'description="\"YOUR_ENRICHMENT_DESCRIPTION\""' \ 
--form 'file=@"PATH_TO_YOUR_ENRICHMENT.CSV_FILE"'

Note:

The PATH_TO_YOUR_ENRICHMENT.CSV_FILE as for example: “/Users/Test/CustomEnrichment.csv”.

Response:

{
    "message": "accepted new enrichment request with id 14",
    "customEnrichmentId": 14
}

Status Codes: 202, 406, 502.

4. Update an existing Custom Enrichment:

Request:

curl --location --request PUT 'https://webapi.coralogix.com/api/v1/external/custom-enrichments/14' \  
--header 'Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \ 
--form 'name="\"YOUR_ENRICHMENT_NAME_V2\""' \ 
--form 'description="\"YOUR_ENRICHMENT_DESCRIPTION\""' \ 
--form 'file=@"PATH_TO_YOUR_ENRICHMENT.CSV_FILE"'

Notes:

• Please take a look at the Update Custom Enrichment URL: https://webapi.coralogix.com/api/v1/external/custom-enrichments/<customEnrichmentID>
  The customEnrichmentID used in the Endpoint (14 in this example), is taken from the initial POST request when the Custom Enrichment to update  was created.
• The PATH_TO_YOUR_ENRICHMENT.CSV_FILE as for example: “/Users/Test/CustomEnrichment_V2.csv”.

Response:

{
    "message": "accepted update to enrichment request with id 14",
    "customEnrichmentId": 14
}

Status Codes: 202, 502.

5. Delete an existing Custom Enrichment:

Request:

curl --location --request DELETE 'https://webapi.coralogix.com/api/v1/external/custom-enrichments/14' \ 
--header 'Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \ --data-raw ''

Note:

• Please take a look at the Delete Custom Enrichment URL: https://webapi.coralogix.com/api/v1/external/custom-enrichments/<customEnrichmentID>
  The customEnrichmentID used in the Endpoint (14 in this example), is taken from the initial POST request when the Custom Enrichment to delete  was created.

Response:

{
    "message": "deleted custom enrichment 14",
    "customEnrichmentId": 14
}

Status Codes: 200, 409, 502.

6. List all existing Custom Enrichments:

Request:

curl --location --request GET 'https://webapi.coralogix.com/api/v1/external/custom-enrichments/' \
--header 'Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'

Response:

[
    {
        "id": 13,
        "name": "Enrichment Test",
        "description": "First Coralogix API Custom Enrichment Test",
        "version": 1
    },
    { 
        "id": 14, "name": "customer's UUID to customer name V2", 
        "description": "This enrichment is for mapping UUID to name", 
        "version": 2 
    }
]

Status Codes: 200, 500.