Learn more about Streama© – the foundational technology behind our stateful streaming data platform. Learn More

Elevate your event data with Custom Data Enrichment in Coralogix

  • Pedro Rodriguez
  • August 17, 2021
custom data enrichment coralogix

Have you ever found yourself late at night combing through a myriad of logs attempting to determine why your cluster went down? Yes, that’s a really stressful job, especially when you think about how much money your company loses as a result of these incidents. Gartner estimates that the revenue lost due to outages is around $5,600/minute, which amounts to more than $330K/hour.

To make matters worse, your boss is breathing down your neck asking for updates, and the logs you are working with look something like this:

non-enriched log event
Not very helpful, huh?

Please, do not despair!  Coralogix Custom Data Enrichment to the rescue!

Using our Custom Data Enrichment feature, you can create one or more “translation” tables that add translated (e.g. enriched) fields to your logs. This allows you to unscramble any obscure data in the fields.

In this example, I created two very simple Custom Enrichments, which would translate the data from the following 2 log fields:

  1. hw-status
  2. errorcode

Please note that the above fields are case-sensitive. I have oversimplified my example to illustrate the essence of our Custom Data Enrichment feature, but in practice, your enrichment .csv files would contain more than one line (up to 10,000 rows, or approximately 0.5 MB of standard information).

Here is the content of the two Custom Data Enrichment .csv files I used for this example:

1. errorcode.csv:

errorcode,translation
%$^&*(#@!!!!!,Pepe unplugged it!

2. hwstatus.csv:

hwstatus,hwstatus-translated
^00^:(,The Power is off

Please note that the header lines in the .csv files are just placeholders (to remind you what kind of data the file contains) and are not referenced during the Custom Enrichment creation. What is important here to keep in mind is that the value to be enriched is the first one in each line, before the second comma-separated value.

Following are the two Custom Enrichments I created for this example:

coralogix enrichment file examples


Please note how the “hw-status”, and “errorcode” fields have been selected.

After creating the Custom Data Enrichment, and sending logs that would match the defined fields in the Enrichments, you will notice new fields added to your logs which are named using the original field’s name, with “_enriched” appended to it. Please take a look at what our example looks like after the Enrichment:

enriched log event coralogix

Isn’t that great? No more guessing what those values are…

We have gathered several .csv Enrichment files that have been used internally, and by our customers, and we are sharing them with you so you could use them to create your own enrichments. 

They are available for download here:

Custom Data EnrichmentDescriptionDownload Link
area-code-to-state.csvMaps US area codes to their corresponding state code(s).Download Now
us-state-or-terrirory-code-to-state-or-country-name.csvMaps US state/territory codes to their corresponding names.Download Now
cisco-cipm-return-codes-to-description.csvMaps Cisco IP Manager return codes
to their corresponding descriptions.
Download Now
country-phone-code-to-country-name.csvMaps country codes to their corresponding country name.Download Now
country-territory-to-currency-symbol-code-unit.csvMaps country names to their corresponding currency names, symbols, and ISO codes.Download Now
country-to-currency-name-code-number-and-minor-unit.csvMaps country names to their corresponding currency and alphabetic/numeric codes.Download Now
cve-code-to-cve-description.csvMaps CVE codes to their corresponding descriptions.Download Now
date-to-day-of-the-week.csvMaps dates to the corresponding day of the week.

*Includes mapping from Friday, July 30th, 2021 to Sunday, February 7th, 2033.
Download Now
gmt-offset-to-civilian-code-during-standard-time.csvMaps GMT Offset codes (ex. +0000) to their corresponding civilian codes (ex. GMT)Download Now
iana-network-protocol-number-to-protocol-name.csvMaps IANA network protocol numbers to their corresponding names.Download Now
iana-network-protocol-number-to-protocol-keyword-to-protocol-name.csvMaps IANA network protocol numbers to their corresponding keywords and names.Download Now
iso-country-code-to-country-name.csvMaps country names to their corresponding short and long ISO Codes.Download Now
naics-industry-codes-to-description.csvMaps NAICS industry codes to their corresponding titles (2017).Download Now
cloudfront-locations.csvMaps IATA Codes to AWS CloudFront CDN Edge Locations.Download Now

Let us know please about any other Custom Enrichments that may be useful and relevant to your environment. If you would like to contribute to our users’ repository with any .csv files that could be used to create new Custom log Enrichments, please send them our way:
support@coralogix.com

The users’ repository will be updated as new .csv files are uploaded. Please check this page frequently. 

We would also love to hear about your experience using the .csv enrichment files, as well as receiving any feedback you may have for us.

For more information about our Custom Data Enrichment feature, check out the complete tutorial here.

Stateful streaming analytics for observability data