Events2Metrics

Open in ChatGPTOpen in ClaudeAPI referenceTerraform

Coralogix Events2Metrics enables generating metrics from your spans and logs to optimize storage without sacrificing important data.

Define a query and Coralogix will execute it every minute and store different data aggregations in a long-term index. Metrics start to gather from the point in time in which they were defined. The available query time range for your Events2Metrics indices is 90 days. With Events2Metrics activated, you can create up to 300 metric rules and set the retention period to any length, providing ample opportunity for data analysis without retention restrictions. Across all your rules, the permutation limits you configure can total up to 10M.

What you need

• To configure Events2Metrics from logs, you need LOGS.EVENTS2METRICS:UPDATECONFIG (Data Admin, Observability Lead, Platform Admin, Security User).
• To configure Events2Metrics from spans, you need SPANS.EVENTS2METRICS:UPDATECONFIG (Data Admin, Observability Lead, Platform Admin).

Enabling Events2Metrics

1. In your Coralogix toolbar, navigate to Data Flow, then Events2Metrics. Select + New Event2Metrics.

2. Fill in the section Details:

   • Name: The name chosen will appear in the field representing this metric in the long-term index.
   • Description: Describe your metric.

3. Select your Event Source by choosing either Logs or Spans.

4. Define your Query. Use a written text query or a Lucene type of query.

5. Define Scope:

   • For Spans: Filter by Applications, Subsystems, and Service. For Actions, filter particular HTTP requests and their associated services.
   • For Logs: Filter by Applications, Subsystems, and Severities.

6. Define Metric Fields (optional).

   Note

   • Define up to 10 fields for which the metrics will be collected.

   • For each metric, you can choose an aggregation function that will aggregate the stream of the data and calculate Max/Min/Count/Avg/Sum/Histogram/Samples in 1 minute granularity.

   • A Histogram bucket is a range of values within a histogram. When you create a histogram, you specify the ranges of the values that will be used to group the data. Each bucket in the histogram represents the number of observations that fall within the range of values for that bucket. More granular buckets provide more accurate percentiles during querying, but will increase the number of time series and storage. A maximum of 30 histograms are permitted.

   • When selecting Histogram aggregation, you will need to provide the buckets that represent the distribution of the data.

   • An example Histogram bucket from a CPU metric looks as follows: 0, 10, 30, 45, 50, 60, 70, 85, 90, 100 - numbers separated by commas.

   • When selecting Samples, you also need to select either Min or Max aggregation. Coralogix collects 4 samples per minute, with each sample representing a quarter of a minute. When we receive data from the same quarter, we take the minimum or the maximum (depending on the selection) from the old and new data.

7. Define Labels: Create up to 10 labels. Contact our support staff should you need more.

8. Choose the maximum amount of Metric Permutations allowed for this metric.

   Note

   • Each metric rule has a default permutation limit. It appears in grey in the input field.
   • The limits you configure across all rules can total up to 10M. That total is checked when you create or edit a rule.
   • A rule that reaches its limit keeps running, and further combinations aggregate into a single overflow series. See What happens at the permutation limit.
   • Given the set Alert conditions above, the graph displays the number of permutations per day that would have theoretically resulted in the previous 7 days for this metric rule for high and medium priority logs. Use this information to determine the maximum amount of permutations you would like to allow for this metric moving forward, based on the theoretical usage. Choose a number that is high enough to allow average daily permutations, but not so high that it will cause you to exceed your overall daily quota.

What happens at the permutation limit

A permutation is one distinct combination of your label values. Each rule registers combinations as it first sees them during the UTC day, up to the limit you configured.

Once a rule reaches its limit, it keeps producing metrics. Every further combination has each of its label values replaced with the constant CoralogixOtherValues, and all of them are aggregated into a single overflow series. The label keys stay, so a rule with labels A and B produces:

metric_name{A="CoralogixOtherValues", B="CoralogixOtherValues"}

Values continue to aggregate into that series, so its counts and sums stay correct in total. The breakdown by label value is what you give up.

For example, a rule with labels A and B and a limit of 50 keeps the first 50 distinct combinations of the day as their own series. Everything after contributes to the one overflow series.

How slots are allocated

Registration is append-only, which has two consequences worth planning around:

• A combination that has already registered keeps its series for the rest of the day. A later arrival cannot replace or evict it.
• A combination that stops reporting partway through the day holds its slot until the day ends.

The first combinations of the day take the slots, whether or not they turn out to be the interesting ones.

The daily cycle

Counting restarts at 00:00 UTC, when new data begins registering against a fresh set of slots for the new day. The previous day's set stays active until 02:00 UTC, so late-arriving data still lands in the day it belongs to.

How the overflow series is counted

Coralogix creates the overflow series after the limit check and never registers it as a combination, so it leaves your slots intact.

Detect that a rule has reached its limit

Coralogix emits no signal when a rule reaches its limit. To find out yourself, query the metric for the overflow value:

metric_name{A="CoralogixOtherValues"}

Data in that series means the rule has passed its limit for the day. Chart it on a dashboard or alert on it to learn at the time rather than afterwards.

The organization quota

The 10M organization figure and the per-rule limit work differently:

	Per-rule permutation limit	10M organization quota
What it is	A runtime behavior	A provisioning budget
When it applies	During ingestion, every day	When you create or edit a rule
What it counts	Combinations actually seen	The limits you have configured, used or not
What happens at the limit	Combinations fold into the overflow series	The rule cannot be saved

Ingestion reads only the per-rule limit. An organization well inside its quota can still have individual rules producing overflow series, and a rule with headroom can still be refused at save time when the configured total would exceed 10M.

When metric generation stops

A rule stops producing metrics in two cases: you delete it, or its definition has not yet propagated. Both limits leave generation running.

Default metrics

With every Events2Metrics created in Coralogix, a default metric is created. This metric counts the total logs or spans that matches the query and the scope that was provided in the Events2Metrics setup. The metric name will be <Events2Metrics Name>_cx_docs_total, and you can query it like any other metric in Metric Explorer or a Custom Dashboard. This default metric can be grouped by the labels that were defined in the Labels section.

Usage

A Count metric is enabled to track the spans or logs that match the Events2Metrics filters.

• In Metric Explorer, query the default metric to see the number of spans or logs that meet the Events2Metrics conditions, then filter or group by the labels defined for the metric. To open it, navigate to Explore, then Metric Explorer.

• You can also utilize PromQL while creating a Metric Alert and query the default metric.

Using PromQL with E2M

• E2M is not a counter or gauge: It emits aggregated values once per minute (for example, sum, min, max over 1m).

• Use _over_time functions: To avoid gaps and get accurate results, use PromQL range functions like sum_over_time, avg_over_time.

• Do not use rate() or increase(): These are not compatible with E2M.

Example

sum(sum_over_time(e2m_cx_sum[5m])) by (app)

Additional resources

• Check out this video tutorial