Safe Use of Recording Rule–Based Metrics in SLOs

Coralogix SLOs can be powered by metrics derived from Prometheus recording rules, which help reduce query complexity and improve performance. However, when those metrics come from chained recording rules—where one rule depends on another—there’s a risk of inaccurate evaluations.

This guide explains:

• Why chained recording rules can cause incomplete or premature SLO results.
• How to safely use recording rule–based metrics in your SLOs.
• How to apply the PromQL offset modifier to ensure evaluation accuracy.

Why chained recording rules can introduce errors

In chained setups, a downstream recording rule might run before its upstream dependency has completed writing data. This timing mismatch can cause the downstream rule to persist incomplete values, which are then used in the SLO. The result: inaccurate or misleading SLO evaluations.

The solution: Apply an offset

To avoid this issue, apply a PromQL offset to your SLO query. The offset shifts the evaluation window backward, ensuring all dependent data is fully materialized before being used.

For example:

increase(my_recorded_metric offset 2m)

This is especially useful in situations where the most recent data might not yet be complete—such as with recording rules, which can have slight delays in updating. Applying an offset (e.g., 2m) helps avoid premature or inaccurate evaluations, particularly in time-sensitive use cases like SLOs.

Query examples

Here are examples showing how to apply the offset properly:

• Basic total events (no filter):

  sum(increase(RR_based_metric)) by (service_name)
  

  With offset:

  sum(increase(RR_based_metric offset 2m)) by (service_name)
  

• Good events (with filter):

  sum(increase(RR_based_metric{status!="error"})) by (service_name)    
  

  With offset:

  sum(increase(RR_based_metric{status!="error"} offset 2m)) by (service_name)   
  

• Compound query 1:

  sum(increase(RR_based_metric_successes)) by (service_name)
  +
  sum(increase(RR_based_metric_failures)) by (service_name)    
  

  With offset:

  sum(increase(RR_based_metric_successes offset 2m)) by (service_name)
  +
  sum(increase(RR_based_metric_failures offset 2m)) by (service_name)    
  

• Compound query 2:

  sum(increase(RR_based_metric_successes) + increase(RR_based_metric_failures)) by (service_name)    
  

  With offset:

  sum(increase(RR_based_metric_successes offset 2m) + increase(RR_based_metric_failures offset 2m)) by (service_name)