Custom evaluation delay

Overview

Lags in your data pipeline can cause delays in log and metric ingestion, potentially leading to false alerts. While alert conditions are evaluated in real-time, delayed data arriving later can retroactively affect whether those conditions are met.

When an alert should be triggered above a user-defined threshold, missing data may delay it or lead to an incorrect resolution. Conversely, when an alert should be triggered below a user-defined threshold, incomplete data can cause a false positive alert or delayed alert resolution.

The Custom Evaluation Delay feature helps mitigate this issue by shifting an alert's evaluation timeframe backward by a configurable amount. This adjustment ensures that alert conditions are evaluated against a more complete dataset, accounting for late-arriving logs or metrics.

By using Evaluation Delay, you can reduce the risk of false positives or negatives caused by real-time data fluctuations and improve the accuracy and reliability of your alerts.

Configuration

Open the alert creation wizard: go to Alerts, then select Create alert. You can also start from a saved query in Explore and select Create alert.
In the Condition step, expand Advanced configurations.
Set Delay alert evaluation to the time period you want the evaluator to wait for late-arriving data.