Custom Data Enrichment

Custom Log Enrichment with Coralogix enables you to easily enrich your log data. Now, you can automatically add fields to your JSON logs based on specific matches in your log data, using a pre-defined custom data source of your own. This way, you can enhance your log data with business, operations, or security information that are not available at runtime.

To get started, first create your CSV file that will define your custom data source. There are 2 supported types of data enrichment, string to string, or string to a JSON object. With string to string the file should include 2 columns: Column A will include a list of values to match and column B will include the enrichment information (the file must include headers although column names have no meaning).  With string to JSON, the file should include at least 3 columns: Column A will include a list of values to match, and columns B, C, etc.. will include the enriched JSON object fields’ information (the file must include headers, which will then be the enriched JSON fields names correspondingly).

Note – file size is limited to 10,000 rows (~0.5MB of standard information).

E.g. a string–>string CSV file that will contain pairs of UUID and customers’ names to allow better readability to the business and support teams.

E.g. a string–>JSON CSV file that will contain in each row a UUID, customer name, and customer size.

Next, navigate to the settings menu –> Enrich and click on the top right blue button ADD CUSTOM ENRICHMENT.

Name your new enrichment and description (optional) and upload your CSV file from earlier.

Now, after your pre-configured enrichment file is uploaded, you can scroll down to the newly added Custom Enrichment section. In this section, you will define fields from your logs whose values should be compared against the CSV file values. When a log record that contains one of these fields with its value matching any value from the CSV file (left column), a new field called field_enriched will be added to the JSON record and its value will be the corresponding value in the right column.

Let’s see how it looks in Coralogix. These are the original logs that are being sent, using the IDs from the CSV files mentioned above.

Using string–>string enrichment:

Using string–>JSON enrichment:

Examples

Monitoring

Assuming we have a log with a UUID that represents a customer, without an actual field that holds the customer name. The goal is to have a field in the log that will contain the customers’ names so we can visualize/search our logs based on them. With custom enrichment, we can create such enrichment by setting up a CSV file to map each UUID to a name.

Security

Given a field with a domain name in our log that represents where our applications were accessed from. We want to create an alert that will notify us if there was any attempt to access our application from an unauthorized domain. For that purpose, we can set a CSV file with a list of the whitelisted domains so that every log with one of these domains will be enriched with a field that will contain the word ‘allowed’, and then we can create an alert for those logs that will not contain this field (e.g. the query will look like: NOT domain_enriched:allowed).

API Support

1. Create the following request to create your new custom enrichment.

POST – https://webapi.coralogix.com/api/v1/external/custom-enrichments

As a bearer token (under authorization) use your Alerts & Rules API Key.

Request
curl --location --request POST 'https://webapi.coralogix.com/api/v1/external/custom-enrichments' \
--header 'Content-Type: application/json' \
--header 'Authorization: bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \
--form 'name=YOUR_ENRICHMENT_NAME' \
--form 'description=YOUR_ENRICHMENT_DESCRIPTION' \
--form 'file=@PATH_TO_ENRICHMENT_CSV_FILE'
Response
{
    "message": "accepted new enrichment request with id 14",
    "customEnrichmentId": 14
}

status codes: 202, 406, 502

2. If you wish to update your custom enrichment, use the following request.

PUT – https://webapi.coralogix.com/api/v1/external/custom-enrichments/{{customEnrichmentId}}

The customEnrichmentId is taken from the initial POST request when creating the custom enrichment.

Request
curl --location --request PUT 'https://webapi.coralogix.com/api/v1/external/custom-enrichments/14' \
--header 'Content-Type: application/json' \
--header 'Authorization: bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \
--form 'name=YOUR_ENRICHMENT_NAME_V2' \
--form 'description=YOUR_ENRICHMENT_DESCRIPTION' \
--form 'file=@PATH_TO_UPDATED_ENRICHMENT_CSV_FILE'
Response
{
    "message": "accepted update to enrichment request with id 14",
    "customEnrichmentId": 14
}

status codes: 202, 502

3. If you wish to remove your custom enrichment, use the following request.

DELETE – https://webapi.coralogix.com/api/v1/external/custom-enrichments/{{customEnrichmentId}}

The customEnrichmentId is taken from the initial POST request when creating the custom enrichment.

Request
curl --location --request GET 'https://webapi.coralogix.com/api/v1/external/custom-enrichments' \
--header 'Content-Type: application/json' \
--header 'Authorization: bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \
Response
{
    "message": "deleted custom enrichment 14",
    "customEnrichmentId": 14
}

status codes: 200, 409, 502

4. If you wish to get all your existing custom enrichments, use the following request.

GET – https://webapi.coralogix.com/api/v1/external/custom-enrichments

Request
curl --location --request GET 'https://webapi.coralogix.com/api/v1/external/custom-enrichments' \
--header 'Content-Type: application/json' \
--header 'Authorization: bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \
Response
[
    {
        "id": 14,
        "name": "customer's UUID to customer name V2",
        "description": "This enrichment is for mapping UUID to name",
        "version": 2
    }
]

status codes: 200, 500