Copy as Markdown[Open in ChatGPT](https://chatgpt.com/?q=Read%20https%3A%2F%2Fcoralogix.com%2Fdocs%2Fuser-guides%2Fdata-flow%2Fstream-aggregation.md%20and%20help%20me%20with%20my%20question%20about%20this%20Coralogix%20documentation%20page.)[Open in Claude](https://claude.ai/new?q=Read%20https%3A%2F%2Fcoralogix.com%2Fdocs%2Fuser-guides%2Fdata-flow%2Fstream-aggregation.md%20and%20help%20me%20with%20my%20question%20about%20this%20Coralogix%20documentation%20page.)

# Stream aggregation

Use Stream Aggregation to aggregate metrics at ingest time, before storage.

Use Stream Aggregation to make high-cardinality metrics usable and reliable at query time, especially when recording rules and if needed, queries cannot scale due to scan and evaluation limits.

By default, Stream Aggregation does not reduce cost. Both the raw metric and the aggregated metric are stored. Reduce storage and query cost only when you explicitly block the raw metric and keep only the aggregated output.

Use Stream Aggregation when raw metrics generate large numbers of unique label combinations that you do not need to store or query individually.

## What you need[​](#what-you-need "Direct link to What you need")

* To view streaming aggregation rules, you need `STREAMING-AGGREGATIONS:READCONFIG` (Data Admin, Observability Lead, Platform Admin, Read-Only User).
* To create or delete streaming aggregation rules, you need `STREAMING-AGGREGATIONS:UPDATECONFIG` (Data Admin, Observability Lead, Platform Admin).

## Why stream aggregation matters[​](#why-stream-aggregation-matters "Direct link to Why stream aggregation matters")

High-cardinality metrics introduce operational challenges:

* Queries and recording rules scan and evaluation limits
* Aggregations become slow or fail to run
* Dashboards and alerts become unreliable under load

Stream Aggregation addresses this by processing metrics in the ingestion path and producing a smaller, stable set of aggregated time series.

This allows you to:

* Avoid creating large numbers of recording rules
* Bypass query scan limits for complex aggregations
* Make high-cardinality data usable for dashboards and alerts

Cost reduction is possible only when Stream Aggregation is combined with blocking the raw metric.

## How stream aggregation differs from recording rules[​](#how-stream-aggregation-differs-from-recording-rules "Direct link to How stream aggregation differs from recording rules")

Stream Aggregation processes metrics during ingestion. Recording rules process metrics after storage.

With Stream Aggregation:

* Aggregation happens once, at ingest time
* Queries run on already-aggregated data
* High-cardinality input does not impact query evaluation

With recording rules:

* Raw samples are stored first
* Each rule must scan all matching series
* High-cardinality metrics can cause rule failures

Select Stream Aggregation when query limits or cardinality make recording rules impractical.

Select recording rules when you need access to raw metrics or historical recomputation.

## Understand the stream aggregation data flow[​](#understand-the-stream-aggregation-data-flow "Direct link to Understand the stream aggregation data flow")

Stream Aggregation runs inside the ingestion pipeline.

### End-to-end data flow[​](#end-to-end-data-flow "Direct link to End-to-end data flow")

1. Metrics arrive at the ingestion layer
2. The system checks whether the metric matches a Stream Aggregation rule
3. If a rule matches, the system sends the samples to the streaming aggregation pipeline
4. The pipeline applies filters, grouping, and aggregation functions
5. The pipeline produces new aggregated metric samples
6. The system sends aggregated samples to the standard metrics pipeline

## Understand aggregation functions and time windows[​](#understand-aggregation-functions-and-time-windows "Direct link to Understand aggregation functions and time windows")

Use aggregation functions to define how incoming samples are combined. Use the time window settings to define how often results are emitted and how much historical data is included.

| **Aggregation function** | **How it aggregates data**                                            | **Evaluation and Lookback behavior**                                                      | **Typical use**                              |
| ------------------------ | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------- |
| **Sum**                  | Adds all sample values in the aggregation window                      | Uses the configured **Lookback** interval and emits results every **Evaluation** interval | Total volume, such as requests or bytes      |
| **Avg**                  | Calculates the average of all sample values in the aggregation window | Uses the configured **Lookback** interval and emits results every **Evaluation** interval | Typical behavior, such as average latency    |
| **Min**                  | Emits the smallest value observed in the aggregation window           | Uses the configured **Lookback** interval and emits results every **Evaluation** interval | Lower bounds and best-case performance       |
| **Max**                  | Emits the largest value observed in the aggregation window            | Uses the configured **Lookback** interval and emits results every **Evaluation** interval | Spikes and outliers                          |
| **Count**                | Emits the number of samples in the aggregation window                 | Uses the configured **Lookback** interval and emits results every **Evaluation** interval | Event frequency and rate analysis            |
| **Sum counter**          | Aggregates counter increases across samples                           | Does not follow the same Evaluation and Lookback behavior as the other functions          | Counter-type metrics where increments matter |

### How the time windows work[​](#how-the-time-windows-work "Direct link to How the time windows work")

Stream Aggregation uses 2 time settings to control how data is aggregated.

**Evaluation interval**

Defines how often the system emits aggregated results.

**Lookback interval**

Defines how much historical data each evaluation includes.

**Example**

* Lookback: 5 minutes
* Evaluation: 1 minute

Every minute, the system aggregates the last 5 minutes of data and writes a new result.

To create tumbling behavior, set the Lookback interval equal to the Evaluation interval.

Note

This Evaluation and Lookback configuration applies to all aggregation functions except **Sum counter**.

## Understand rule lifecycle and immutability[​](#understand-rule-lifecycle-and-immutability "Direct link to Understand rule lifecycle and immutability")

Stream Aggregation rules follow a strict lifecycle:

* Rules are created in an enabled state
* Once you disable a rule, it cannot be enabled again

Disabling a rule permanently stops aggregation and prevents accidental reactivation of incorrect or costly configurations. Create a new rule if you need to restart aggregation with changes.

## Open stream aggregation[​](#open-stream-aggregation "Direct link to Open stream aggregation")

[![stream aggregation create rule](/docs/assets/images/stream-aggregation-47af83bc8f54f1530b3ff36861afe15f.webp)](https://coralogix.com/docs/assets/images/stream-aggregation-47af83bc8f54f1530b3ff36861afe15f.webp)

Manage Stream Aggregation from the Data Flow area.

1. Go to **Data Flow**
2. Under **Metrics**, select **Stream Aggregation**

The Stream Aggregations page lists all rules, their configuration, and their current state.

## Create a stream aggregation rule[​](#create-a-stream-aggregation-rule "Direct link to Create a stream aggregation rule")

1. Select **Add new rule**

2. In **Policy name**, enter a descriptive name

3. (Optional) In **Description**, describe the purpose of the rule

4. In **Source metric**, select the metric you want to aggregate

5. In **Output metric name**, enter the name for the aggregated metric

6. (Optional) In **Filters**, add attributes to limit which series the rule processes

7. In **Group by**, select the labels you want to retain in the output

8. In **Time window**, set:

   <!-- -->

   * **Evaluation**
   * **Lookback**

9. In **Aggregation**, select 1 or more aggregation functions

10. Select **Add**

The system creates the rule in a disabled state.

## Disable a stream aggregation rule[​](#disable-a-stream-aggregation-rule "Direct link to Disable a stream aggregation rule")

1. Open the rule
2. Toggle **Enable rule** off

Once disabled, the rule cannot be enabled again. The system retains the rule for audit and reference but no longer processes data.

Create a new rule if you need to resume aggregation.

## Control cardinality with group by[​](#control-cardinality-with-group-by "Direct link to Control cardinality with group by")

The **Group by** setting determines which labels remain in the output metrics.

* The system drops all labels that you do not include
* Each unique combination of retained labels creates 1 output series

Use the smallest set of labels that still answers your monitoring questions.

## Understand output metrics[​](#understand-output-metrics "Direct link to Understand output metrics")

Define the output metric name when you create a Stream Aggregation rule.

The system no longer enforces a fixed naming pattern. The name you enter in **Output metric name** becomes the stored metric name.

Each aggregated metric includes the label:

```
cx_source="stream_aggr"
```

Use this label to identify metrics generated by Stream Aggregation.

## Know current limitations[​](#know-current-limitations "Direct link to Know current limitations")

Stream Aggregation has known limitations:

* No deduplication of incoming samples
* Possible double counting in high-availability setups
* No historical backfill
* Lookback limited to 5 minutes
* Accuracy target of 99 %

Use Stream Aggregation for trend detection and operational monitoring rather than exact billing or compliance use cases.

## Avoid common mistakes[​](#avoid-common-mistakes "Direct link to Avoid common mistakes")

Design Stream Aggregation rules carefully to prevent incorrect results, unstable queries, or unnecessary cost.

* **Including unnecessary labels in Group by**

  Keeping labels that you do not need increases output cardinality and reduces the benefit of aggregation. Only retain labels that are required for analysis or alerting.

* **Using long lookbacks with short evaluation intervals when you expect tumbling behavior**

  This configuration produces overlapping windows rather than discrete time buckets. If you need tumbling aggregation, set the Lookback interval equal to the Evaluation interval.

* **Disabling a rule expecting to reenable it later**

  Once disabled, a rule cannot be enabled again. Create a new rule if you need to change or restart aggregation.

Plan rules carefully before enabling them.