Logs2Metrics

Note! Coralogix Events2Metrics supersedes this current page. New customers wishing to generate metrics from spans and logs should visit the Events2Metrics page to get started. We strongly encourage existing customers to migrate to Events2Metrics as well.

Coralogix Logs2Metrics enables you to generate metrics from your log data to optimize storage without sacrificing important data. You simply define a query and Coralogix will execute it every minute and store different data aggregations in a long-term index. Metrics start to gather from the point in time in which they were defined. The available query time range for your Logs2Metrics indices is 90 days.

Activating Logs2Metrics allows you to create up to 30 metrics with a 12 months retention period.

Guide

1. In Coralgix, Go to the Data Flow tab –> Logs2Metrics section and click on the New Metric button.
   
   
2. Define your metric:
   Metric Name – The name you will choose will be the name of the field representing this metric in the long-term index and will be used in dashboard visualizations.
   Metric Description – Describe your Metric.
   Search Query – Use a written text query or a Lucene type of query (similar to Logs/alerts queries). For example,  coralogix.metadata.subsystemName:nginx* will store the count of logs from any subsystem that starts with Nginx every minute which would allow long-term analysis on your Nginx subsystem. You can also set filters on applications, subsystems, and severities. This is just an example. The query section can have anything you want to have for instance you can put status.numeric:[500 TO 599] which will store data about status code 5xx.
3. “Metric Fields” (optional) –
   Define the fields the metrics data will be collected for, you can define up to 10 fields. These fields will be available to define and chart the metrics.
   
   In the above example we defined a metric field for our log2metrics. The metric field name is response_time. Coralogix will collect aggregated metrics values for the field response_time and will keep them under the following field names, within the log2metrics index:
   metrics.response_time.avg
   metrics.response_time.count
   metrics.response_time.max
   metrics.response_time.min
   metrics.response_time.sum
   Another field that will be created is metrics.response_time.value. It holds an array of the actual response_time values from the logs aggregated by the defined log2metric.
   
4. “Labels” (Optional) –
   Define the Logs2Metric labels, used to visualize your metric data. By default, you can create up to 6 labels. Contact support for more if needed.
   
   
   You may use any mapped field as a label, depending on what you are trying to do. Also, you can give the labels any name you may like as long as it makes sense to you.

Note:

When creating a Logs2Metrics that expects a list of labels, logs that do not include some of the labels are converted into metric documents that only have a subset of the expected labels. Note that when using PromQL with Grafana variables, there is no option to retrieve metrics with a subset of the labels, returning only a partial subset of the results.

See screen shot below.

This behavior has been rectified by populating the missing labels with data and have the correct results showing.

The “Default Metric“

With every L2M created in Coralogix, regardless to specific “Metric Fields” and “Labels” you may define, a default metric is created. This metric will be enabled with the L2M creation in Grafana and when utilizing PromQL. The “Default metric” label will be structured as “<The Metric Name>_cx_docs_total” e.g.: The L2M name is: “Status_500”, then the “Default Metric” name will automatically set as: “Status_500_cx_docs_total”.

Usage:

A “Count” metric is enabled for tracking the number of logs matches the L2M filters,

i.e.: in Grafana, you can explore the number of logs that meet the L2M conditions such as:

– Query

– Applications

– Subsystems

– Severities

You may also utilize PromQL (while creating a “Metric Alert”) and query the “Default Metric”,

e.g.: “sum(Status_500_cx_docs_total)”.

Metrics Permutations limit

A Logs2metric labels permutations (the unique combination of each of the labels values) is a finite number and it is defined by the user per Metric (at the bottom of the metric definition). By default, it is set to 30,000 permutations and you may choose different max permutations value per Metric, while the maximum permutation per account is 1,000,000.

At the top of your defined Metrics list, you will see how many available permutations you have left for your disposal.

Normally, a metric document will show the count of logs per unique permutation of the chosen labels. If you encounter a metric that shows under permutations amount an exclamation mark near the number of allocated permutations it means you have reached the permutations limit. You should adjust the permutation allocation to accommodate all possible permutations, otherwise, the metric document will contain an aggregated log count under a CoralogixOtherValues bucket.

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].