Custom Data Enrichment

Custom Log Enrichment with Coralogix enables you to easily enrich your log data. Now you can automatically add fields to your JSON logs based on specific matches in your log data, using a pre-defined custom data source of your own. This allows you to enhance your log data with business, operations, or security information that is not available at runtime.

To get started, first create your CSV file that will define your custom data source. There are 2 supported types of data enrichment, string to string, or string to a JSON object. With string to string the file should include 2 columns: Column A will include a list of values to match and column B will include the enrichment information (the file must include headers although column names have no meaning).  With string to JSON, the file should include at least 3 columns: Column A will include a list of values to match, and columns B, C, etc.. will include the enriched JSON object fields’ information (the file must include headers, which will then be the enriched JSON fields names correspondingly).

Note – file size is limited to 10,000 rows (~0.5MB of standard information).

E.g. a string–>string CSV file that will contain pairs of UUID and customers’ names to allow better readability to the business and support teams.

E.g. a string–>JSON CSV file that will contain in each row a UUID, customer name, and customer size.

Once the CSV of your choice is ready, navigate to the Data Flow top menu –> Data Enrichment and click on the top right blue button ADD CUSTOM ENRICHMENT.

Name your new enrichment and description (optional) and upload your CSV file from earlier.

Now, after your pre-configured enrichment file is uploaded, you can scroll down to the newly added Custom Enrichment section. In this section, you will define fields from your logs whose values should be compared against the CSV file values. When a log record contains one of these fields with its value matching any value from the CSV file (first column), a new field called field_enriched will be added to the JSON record and its value will be the corresponding value in the next column(s).

Let’s see how it looks in Coralogix. These are the original logs that are being sent, using the IDs from the CSV files mentioned above.

Using string–>string enrichment:

Using string–>JSON enrichment:

Examples

Monitoring

Assuming we have a log with a UUID that represents a customer, without an actual field that holds the customer name. The goal is to have a field in the log that will contain the customers’ names so we can visualize/search our logs based on them. With custom enrichment, we can create such enrichment by setting up a CSV file to map each UUID to a customer name.

Security

Given a field with a domain name in our log that represents where our applications were accessed from. We want to create an alert that will notify us if there was any attempt to access our application from an unauthorized domain. For that purpose, we can set a CSV file with a list of the whitelisted domains so that every log with one of these domains will be enriched with a field that will contain the word ‘allowed’, and then we can create an alert for those logs that will not contain this field (e.g. the query will look like: NOT domain_enriched:allowed).

API Support

1. Please follow these steps to create a Custom Enrichment using the Coralogix API:

• Make sure to select the correct API endpoint for your cluster:

Cluster	API Endpoint
.us	https://webapi.coralogix.us/api/v1/external/custom-enrichments
.in	https://api.app.coralogix.in/api/v1/external/custom-enrichments
.com	https://webapi.coralogix.com/api/v1/external/custom-enrichments
eu2.coralogix.com	https://webapi.eu2.coralogix.com/api/v1/external/custom-enrichments
coralogixsg.com	https://webapi.coralogixsg.com/api/v1/external/custom-enrichments

• Use the Alerts, Rules and Tags API Key as the Bearer token.
  (Click on the Data Flow top menu –> API Keys. [If you have not yet generated the key, please click “GENERATE NEW API KEY”]. Copy the key there).

2. The following Custom Enrichments API calls are supported:

• Creation
• Update
• Deletion
• Listing

3. Create a new Custom Enrichment (copy the curl commands below and customize them for your environment before sending them to the API. Please remember to use the correct endpoint for your deployment as described earlier in this tutorial):

Request:

curl --location --request POST 'https://webapi.coralogix.com/api/v1/external/custom-enrichments' \ 
--header 'Content-Type: application/json' \ 
--header 'Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \ 
--form 'name="\"YOUR_ENRICHMENT_NAME\""' \ 
--form 'description="\"YOUR_ENRICHMENT_DESCRIPTION\""' \ 
--form '[email protected]"PATH_TO_YOUR_ENRICHMENT.CSV_FILE"'

Note:

The PATH_TO_YOUR_ENRICHMENT.CSV_FILE as for example: “/Users/Test/CustomEnrichment.csv”.

Response:

{
    "message": "accepted new enrichment request with id 14",
    "customEnrichmentId": 14
}

Status Codes: 202, 406, 502.

4. Update an existing Custom Enrichment:

Request:

curl --location --request PUT 'https://webapi.coralogix.com/api/v1/external/custom-enrichments/14' \ 
--header 'Content-Type: application/json' \ 
--header 'Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \ 
--form 'name="\"YOUR_ENRICHMENT_NAME_V2\""' \ 
--form 'description="\"YOUR_ENRICHMENT_DESCRIPTION\""' \ 
--form '[email protected]"PATH_TO_YOUR_ENRICHMENT.CSV_FILE"'

Notes:

• Please take a look at the Update Custom Enrichment URL: https://webapi.coralogix.com/api/v1/external/custom-enrichments/<customEnrichmentID>
  The customEnrichmentID used in the Endpoint (14 in this example), is taken from the initial POST request when the Custom Enrichment to update  was created.
• The PATH_TO_YOUR_ENRICHMENT.CSV_FILE as for example: “/Users/Test/CustomEnrichment_V2.csv”.

Response:

{
    "message": "accepted update to enrichment request with id 14",
    "customEnrichmentId": 14
}

Status Codes: 202, 502.

5. Delete an existing Custom Enrichment:

Request:

curl --location --request DELETE 'https://webapi.coralogix.com/api/v1/external/custom-enrichments/14' \ 
--header 'Content-Type: application/json' \ 
--header 'Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \ --data-raw ''

Note:

• Please take a look at the Delete Custom Enrichment URL: https://webapi.coralogix.com/api/v1/external/custom-enrichments/<customEnrichmentID>
  The customEnrichmentID used in the Endpoint (14 in this example), is taken from the initial POST request when the Custom Enrichment to delete  was created.

Response:

{
    "message": "deleted custom enrichment 14",
    "customEnrichmentId": 14
}

Status Codes: 200, 409, 502.

6. List all existing Custom Enrichments:

Request:

curl --location --request GET 'https://webapi.coralogix.com/api/v1/external/custom-enrichments/' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'

Response:

[
    {
        "id": 13,
        "name": "Enrichment Test",
        "description": "First Coralogix API Custom Enrichment Test",
        "version": 1
    },
    { 
        "id": 14, "name": "customer's UUID to customer name V2", 
        "description": "This enrichment is for mapping UUID to name", 
        "version": 2 
    }
]

Status Codes: 200, 500.