Heroku Metric Logs
Our Heroku integration is configured to automatically detect inbound platform and custom metric logs, and parse them into JSON. To avoid confusion, we'll show some before and after examples, as well as explaining how to parse and generate metrics from these values, once they appear in your account.
Before & after
Heroku metric logs appear as key-value pairs, in Syslog format. Coralogix automatically extracts the Syslog metadata, and makes the body of the syslog message available.
The unparsed body of the log (without syslog metadata) appears like this:
dyno=heroku.7af55a6f-b5eb-46e7-b05c-36d3dc04ef22.5df262fa-c95f-4e06-a6c1-6d189b9781e9 source=web.1 sample#load_avg_1m=0.05 sample#load_avg_5m=0.03 sample#load_avg_15m=0.02
While this is readable, it is difficult to query each individual field without complex regex parsing. To make this easier for our Heroku customers, we have enabled JSON transformation for this incoming data, so that when it arrives in your Coralogix account, the above log body is transformed into this:
{"dyno":"heroku.7af55a6f-b5eb-46e7-b05c-36d3dc04ef22.5df262fa-c95f-4e06-a6c1-6d189b9781e9","sample#load_avg_15m":"0.02","sample#load_avg_1m":"0.05","sample#load_avg_5m":"0.03","source":"web.1"}
This means that, rather than extracting the values you need, customers can run direct queries on their incoming metric logs, for example, the following query can extract all metric samples where the number of active connections are greater than 0:
Of course, this also opens up reporting for Heroku customers too, who can build queries that compute average over time, using DataPrime:
filter addon != null
| groupby addon avg($d['sample#memory-percentage-used']:num) as avg_memory_utilization
I am not seeing JSON parsed metrics
This feature has been enabled for new users only, to ensure backwards compatibility with our existing Heroku users. Contact our support team via the in-app chat if you wish to move to the new format. In time, this will be rolled out for all customers.
Handling of units
Units are automatically extracted from the value of the metric, and suffixed into the key. For example, consider the following metric log, which comes from Heroku:
source=DATABASE addon=postgresql-animate-79954 ... sample#memory-total=3944456kB sample#memory-free=84080kB sample#memory-percentage-used=0.97868 sample#memory-cached=3194564kB
Notice that the value of sample#memory-total is 3944456kB. Our integration attempts to ensure that this is a calculable numeric value out of the box, so it converts this into sample#memory-total-kB with value 3944456. This means that it is natively available for conversion using Events2Metrics.
Note that any pattern after the numeric value is considered a unit, and will be handled in this way, for example: sample#my-metric=100foo will become {"sample#my-metric-foo": 100}. This will also preserve the unit casing.
This may cause edge cases if undesired special characters appear as units in your logs. In this case, you should use Parsing Rules to change the value. See below for an example of how to handle these edge cases using parsing rules.
Converting incoming metric logs into metrics
For long term retention, Coralogix works well for logs, but to maintain high performance over longer periods of time, conversion to metrics provides a low cost, simple method for constantly querying data over months. For example, if we consider the sample#active-connections field, we can define an Events2Metric for this, that will automatically generate a timeseries metric for this field:
This practice is especially useful for users who wish to maintain dashboard performance over long periods of time for a series of important metrics. It is also important to label your metrics with useful dimensions on which you wish to query. For example, in the above example, we also extract a number of fields that function as labels:
This means that data can now be queried using PromQL in the Coralogix app, and the labels will be available on the metric. For example, the following query shows Postgres Read iOps over time, grouped by source & addon names:
avg(postgres_read_iops{postgres_addon=~'{{ postgres_addon }}',postgres_source=~'{{ postgres_source }}', postgres_application=~'{{ postgres_application }}'}) by (postgres_source, postgres_addon) or on() vector(0)
Custom application metrics
Custom application metrics that conform to the expected format will also be converted. That is, they should follow <metric-type>#<metric-name>=<value>. For example, counter#total_cpu_seconds=500. If you wish for your metrics to be available for conversion into Coralogix metrics using Events2Metrics, then the value MUST be numeric or decimal followed by a unit.
Custom application metrics embedded in other logs
Currently, custom metrics that are embedded inside other logs do not currently work. For example, if your application writes something like:
This will not be parsed currently. This is because our engine is looking for a key value pattern in the log body, and while this contains a key value pattern, it is not entirely composed of them. For this reason, it will not be parsed. See the section entitled Handling embedded metrics below for a guide on how to solve this problem.
Handling unit edge cases with parsing rules
If we consider the following incoming log that we want to extract:
This will be converted into JSON by our integration, so what arrives at your Coralogix account will look like this:
In its current form, this is valid JSON and will work just fine, although you may wish to alter this to be more readable. To do this, navigate to Data Flow > Parsing Rules.
From here, either add a rule to an existing rule group or select Replace from the rule options at the top of the page:
Then using the following Regular expression, we can replace the value with the desired suffix.
For easy editing, the regular expression is simple -% (the value we're looking to replace) and the replacement string is -perc, which is the replacement value. This will transform the above log into:
Handling embedded metrics
If we consider the following incoming log:
We can see that within this log is our metric, but our integration engine will not yet automatically parse this value. In order to capture this value, we must use parsing rules to bring this metric out into a JSON key-value pair. Access Data Flow > Parsing Rules, and select the extract parsing rule:
To extract your desired value, it's very simple. You simply need to isolate the value and declare the key name using a regex capture group, like so:
We can see in the log, the name of the key we're interested in is counter#total_cpu_seconds. We also know that the value is entirely numeric, which will almost always be the case because these are metrics.
This is why at the start of our regular expression, we have the key name. This is informing our regex where to look in the string. We then have =, which is the character immediately before the value. We have a named capture group called total_cpu_seconds (which will become the name of the resultant JSON key) and this capture group is looking for \d+ which means one or more numerical values.
Handling embedded metrics with units
The idiomatic method for manually handling custom metrics with units is to parse the metric in the same way, but suffix the name of the resultant key with the unit. For example, considering the following input log:
A simple way to preserve the unit while ensuring the value is numerical is:
Note that -kb has been suffixed onto the end of the metric.
Handling multiple metrics in a single embedded log
Parsing rules can capture multiple metrics. If you know the expected order of your metrics (i.e they're always written in the same order) then handling them is simple:
counter#total_cpu_seconds=(?<total_cpu_seconds>\d+) sample#memory_usage=(?<sample_memory_usage-kb>\d+)