Learn more about Streama© – the foundational technology behind our stateful streaming data platform. Learn More

Custom Data Enrichment

Custom Log Enrichment with Coralogix enables you to easily enrich your log data. Now, you can automatically add fields to your JSON logs based on specific matches in your log data, using a pre-defined custom data source of your own. This way, you can enhance your log data with business, operations, or security information that are not available at runtime.

To get started, first create your CSV file that will define your custom data source. There are 2 supported types of data enrichment, string to string, or string to a JSON object. With string to string the file should include 2 columns: Column A will include a list of values to match and column B will include the enrichment information (the file must include headers although column names have no meaning).  With string to JSON, the file should include at least 3 columns: Column A will include a list of values to match, and columns B, C, etc.. will include the enriched JSON object fields’ information (the file must include headers, which will then be the enriched JSON fields names correspondingly).

Note – file size is limited to 10,000 rows (~0.5MB of standard information).

E.g. a string–>string CSV file that will contain pairs of UUID and customers’ names to allow better readability to the business and support teams.

E.g. a string–>JSON CSV file that will contain in each row a UUID, customer name, and customer size.

Next, navigate to the settings menu –> Enrich and click on the top right blue button ADD CUSTOM ENRICHMENT.

Name your new enrichment and description (optional) and upload your CSV file from earlier.

Now, after your pre-configured enrichment file is uploaded, you can scroll down to the newly added Custom Enrichment section. In this section, you will define fields from your logs whose values should be compared against the CSV file values. When a log record that contains one of these fields with its value matching any value from the CSV file (left column), a new field called field_enriched will be added to the JSON record and its value will be the corresponding value in the right column.

Let’s see how it looks in Coralogix. These are the original logs that are being sent, using the IDs from the CSV files mentioned above.

Using string–>string enrichment:

Using string–>JSON enrichment:

Examples

Monitoring

Assuming we have a log with a UUID that represents a customer, without an actual field that holds the customer name. The goal is to have a field in the log that will contain the customers’ names so we can visualize/search our logs based on them. With custom enrichment, we can create such enrichment by setting up a CSV file to map each UUID to a name.

Security

Given a field with a domain name in our log that represents where our applications were accessed from. We want to create an alert that will notify us if there was any attempt to access our application from an unauthorized domain. For that purpose, we can set a CSV file with a list of the whitelisted domains so that every log with one of these domains will be enriched with a field that will contain the word ‘allowed’, and then we can create an alert for those logs that will not contain this field (e.g. the query will look like: NOT domain_enriched:allowed).

API Support

1. Please follow these steps to create a Custom Enrichment using the Coralogix API:

  • Make sure to select the correct API endpoint for your cluster:
ClusterAPI Endpoint
.comhttps://webapi.coralogix.com/api/v1/external/custom-enrichments
.ushttps://webapi.coralogix.us/api/v1/external/custom-enrichments
.inhttps://api.app.coralogix.in/api/v1/external/custom-enrichments
  • Use the Alerts, Rules and Tags API Key as the Bearer token.
    (Click your account’s icon on the top-right of the Coralogix UI, followed by “Settings”, and “API Access” on the left, [If you have not yet generated the key, please click “GENERATE NEW API KEY”]. Copy the key from here).

2. The following Custom Enrichments API calls are supported:

  • Creation
  • Update
  • Deletion
  • Listing

3. Create a new Custom Enrichment (copy the curl commands below and customize them for your environment before sending them to the API. Please remember to use the correct endpoint for your deployment as described earlier in this tutorial):

<div>
<p><strong>Request:</strong></p>
<pre>curl --location --request <strong>POST</strong> 'https://webapi.coralogix.com/api/v1/external/custom-enrichments' \ 
--header 'Content-Type: application/json' \ 
--header 'Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \ 
--form 'name="\"YOUR_ENRICHMENT_NAME\""' \ 
--form 'description="\"YOUR_ENRICHMENT_DESCRIPTION\""' \ 
--form 'file=@"PATH_TO_YOUR_ENRICHMENT.CSV_FILE"'</pre>
<p><em><strong>Note:</strong> </em></p>
<ul>
<li><em>The PATH_TO_YOUR_ENRICHMENT.CSV_FILE as for example: "/Users/Test/CustomEnrichment.csv".</em></li>
</ul>
</div>
<div class="highlight highlight-source-json">
<p><strong>Response:</strong></p>
</div>
<div class="highlight highlight-source-json">
<pre>{
    "message": "accepted new enrichment request with id 14",
    "customEnrichmentId": 14
}</pre>
</div>

Status Codes: 202, 406, 502.

4. Update an existing Custom Enrichment:

Request:

curl --location --request PUT 'https://webapi.coralogix.com/api/v1/external/custom-enrichments/14' \ 
--header 'Content-Type: application/json' \ 
--header 'Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \ 
--form 'name="\"YOUR_ENRICHMENT_NAME_V2\""' \ 
--form 'description="\"YOUR_ENRICHMENT_DESCRIPTION\""' \ 
--form 'file=@"PATH_TO_YOUR_ENRICHMENT.CSV_FILE"'

Notes:

  • Please take a look at the Update Custom Enrichment URL: https://webapi.coralogix.com/api/v1/external/custom-enrichments/<customEnrichmentID>
    The customEnrichmentID used in the Endpoint (14 in this example), is taken from the initial POST request when the Custom Enrichment to update  was created.
  • The PATH_TO_YOUR_ENRICHMENT.CSV_FILE as for example: “/Users/Test/CustomEnrichment_V2.csv”.

Response:

<div class="highlight highlight-source-json">
<pre>{
    "message": "accepted update to enrichment request with id 14",
    "customEnrichmentId": 14
}</pre>
</div>

Status Codes: 202, 502.

5. Delete an existing Custom Enrichment:

Request:

<div>
<pre>curl --location --request <strong>DELETE</strong> 'https://webapi.coralogix.com/api/v1/external/custom-enrichments/14' \ 
--header 'Content-Type: application/json' \ 
--header 'Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \ --data-raw ''</pre>
<p><strong><em>Note:</em></strong></p>
</div>
  • Please take a look at the Delete Custom Enrichment URL: https://webapi.coralogix.com/api/v1/external/custom-enrichments/<customEnrichmentID>
    The customEnrichmentID used in the Endpoint (14 in this example), is taken from the initial POST request when the Custom Enrichment to delete  was created.

Response:

<div class="highlight highlight-source-json">
<pre>{
    "message": "deleted custom enrichment 14",
    "customEnrichmentId": 14
}</pre>
</div>

Status Codes: 200, 409, 502.

6. List all existing Custom Enrichments:

Request:

<div class="highlight highlight-source-json">
<div>
<pre>curl --location --request <strong>GET</strong> 'https://webapi.coralogix.com/api/v1/external/custom-enrichments/' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'</pre>
</div>
</div>

Response:

[
{
"id": 13,
"name": "Enrichment Test",
"description": "First Coralogix API Custom Enrichment Test",
"version": 1
},
{
"id": 14, "name": "customer's UUID to customer name V2",
"description": "This enrichment is for mapping UUID to name",
"version": 2
}
]

Status Codes: 200, 500.