Copy as Markdown[Open in ChatGPT](https://chatgpt.com/?q=Read%20https%3A%2F%2Fcoralogix.com%2Fdocs%2Fuser-guides%2Fslos%2Fbest-practices.md%20and%20help%20me%20with%20my%20question%20about%20this%20Coralogix%20documentation%20page.)[Open in Claude](https://claude.ai/new?q=Read%20https%3A%2F%2Fcoralogix.com%2Fdocs%2Fuser-guides%2Fslos%2Fbest-practices.md%20and%20help%20me%20with%20my%20question%20about%20this%20Coralogix%20documentation%20page.) # Design reliable SLOs An SLO is a measurement system built from several moving parts: the target, time frame, SLI type, query shape, grouping, traffic volume, and alerting strategy all have to fit the service. When they don't, the SLO still produces numbers — they just stop meaning what you expect. This guide explains how to design SLOs that stay stable and operationally useful, and how to recognize when an SLO is the wrong tool. It builds on [Create Service Level Objectives](https://coralogix.com/docs/docs/user-guides/slos/create/.md), which covers the creation workflow itself. Most surprising SLO behavior comes from a small set of causes: * Too little data for the chosen target. * Too many grouping permutations, which splits the data into tiny per-label SLOs. * Sparse or intermittent time series. * Rolling-window effects, where old good or bad data enters or leaves the window. * Request spikes changing the denominator for event-based SLOs. * Latency percentiles measured over too few requests. * Alerting on an SLO that a direct metric, log, or trace alert would handle better. ## Match the SLO type to the signal[​](#match-the-slo-type-to-the-signal "Direct link to Match the SLO type to the signal") Coralogix supports three [SLO types](https://coralogix.com/docs/docs/user-guides/slos/create/.md). Start from the question you want the SLO to answer, then pick the type that fits. | Question | Best SLO type | | ------------------------------------------------------------ | --------------------------------------------------- | | What percentage of requests succeeded? | APM or event-based | | What percentage of requests were under a latency threshold? | Event-based, if histogram buckets or counts exist | | Was latency or error rate below a threshold in most windows? | Time window | | Did the batch job run successfully? | Usually a direct alert; an optional reporting SLO | | Did expected traffic disappear? | A no-data or absence alert, not a request-ratio SLO | For instrumented services already in [APM](https://coralogix.com/docs/docs/user-guides/apm/features/service-slos/.md), an APM SLO is the best default — Coralogix generates the underlying PromQL for you. ## Understand what the error budget measures[​](#understand-what-the-error-budget-measures "Direct link to Understand what the error budget measures") Every SLO reduces to two counts: **good** (events or windows that met the objective) and **total** (all events or evaluated windows). Compliance is `good / total`, and the error budget is how much of the allowed badness you have consumed. Two percentages describe the same SLO, and they are easy to confuse: * **Observed bad rate** — bad events as a percentage of all events. * **Error budget used** — how much of the allowed badness has been consumed. For a 99% target over 10,000 requests with 52 failures: * Allowed bad requests: 1% of 10,000 = 100 * Observed bad rate: 52 / 10,000 = 0.52% * Error budget used: 52 / 100 = 52% * Remaining error budget: 100% − 52% = 48% So the accurate phrasing is: the service had a 0.52% observed error rate, which consumed 52% of the allowed error budget, leaving 48% remaining. Status follows the remaining budget, not raw compliance. | Remaining error budget | Status | | ---------------------- | -------- | | 75%–100% | OK | | 25%–<75% | Warning | | >0%–<25% | Critical | | 0% | Breached | A key consequence: a 99% SLO running at exactly 99% compliance has used all of its error budget. It sits at the floor, not in a healthy state. For the full calculation walkthrough per SLO type, see [SLO calculation example](https://coralogix.com/docs/docs/user-guides/slos/create/.md#slo-calculation-example). ## Give the SLO enough data[​](#give-the-slo-enough-data "Direct link to Give the SLO enough data") Low traffic is the most common reason an SLO looks wrong. Because compliance is `good / total`, the smallest non-zero error rate an SLO can represent is `1 / total`. When that is larger than the error budget, a single bad event breaches the SLO. For a 99% SLO with 50 requests in the window and one failure: * Allowed bad requests: 1% of 50 = 0.5 * Actual bad requests: 1 * Error budget used: 1 / 0.5 = 200% * Remaining error budget: 0% — breached This does not mean the calculation is wrong. It means the target is too strict for the available data. [![Bar chart showing one bad event consumes 200% of budget at 50 events but 1% at 10,000 events](/docs/assets/images/budget-vs-volume-c7576fe5fe152b4bd95b2027e58348c7.webp)](https://coralogix.com/docs/docs/assets/images/budget-vs-volume-c7576fe5fe152b4bd95b2027e58348c7.webp) Shows how much of a 99% error budget a single bad event consumes as total volume grows — target choice must follow volume. To survive one bad event without breaching, you need `total_events_in_window > 1 / (1 - target)`. To stay OK after one bad event, you need `total_events_in_window >= 1 / (0.25 × (1 - target))`, because OK requires at least 75% of the budget remaining. | Target | Error budget | Events to survive 1 bad event | Events to stay OK after 1 bad event | | ------ | ------------ | ----------------------------- | ----------------------------------- | | 95% | 5% | >20 | ≥80 | | 99% | 1% | >100 | ≥400 | | 99.9% | 0.1% | >1,000 | ≥4,000 | | 99.99% | 0.01% | >10,000 | ≥40,000 | These numbers are per SLO window and per permutation. If an SLO groups by `service`, `route`, and `region`, run the volume check for each resulting combination, not for the whole service. Use the volume per permutation to decide whether an SLO is even the right tool: | Volume per permutation per window | Create an SLO? | SLO alerting | Direct alerting | | --------------------------------- | ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------ | | 0 events | No useful request-ratio SLO. No data is not proof of health. | Do not use SLO alerts. | Use a no-data, absence, or heartbeat alert if traffic is expected. | | 1–99 events | Usually no, unless it is clearly reporting-only. One event can dominate the budget. | Avoid. | Use direct alerts for specific failures. | | 100–399 events | Maybe, with a loose target and minimal grouping. 99% can still breach from a few errors. | Prefer only an Error budget alert, if any. | Use direct alerts for urgent failures. | | 400–3,999 events | Yes for 99% if the SLI is clear. Usually not enough for 99.9%. | Error budget alerts are reasonable; Burn rate alerts only after testing. | Keep direct alerts for hard outages and no-data. | | 4,000–39,999 events | Yes. 99.9% can be meaningful if behavior supports it. | Error budget and Burn rate alerts can work if the signal is stable. | Still needed for immediate failures. | | 40,000+ events | Yes. Higher targets can be reasonable. | SLO-based alerting is usually viable, if grouping is controlled. | Use for hard failures, saturation, absence, and runbook-triggering conditions. | ## Control grouping[​](#control-grouping "Direct link to Control grouping") Grouping creates one independently evaluated permutation per label-value combination. That is useful when ownership and behavior genuinely differ, but it also splits the data. A service with 100,000 requests over 28 days averages 1,000 requests per route across 100 routes — and only 200 requests per permutation once you add five regions. A service-level 99.9% SLO that looks safe at 100,000 requests can become meaningless split across 500 permutations. To keep grouping healthy: * Group only by bounded, low-cardinality labels — `service`, `environment`, `region`, a controlled endpoint group, or team. * Avoid high-cardinality labels — raw URL, user ID, customer ID, request ID, pod UID, or trace ID. * Split a high-value tenant into its own filtered SLO rather than one large grouped SLO. * Keep the expected permutation count well under the SLO permutation limit. Ownership tags assign Service, Environment, and Team to permutations, but they do not create permutations — the query grouping does. See [Tag SLO permutations with ownership](https://coralogix.com/docs/docs/user-guides/slos/create/.md#step-5-tag-slo-permutations-with-ownership). ## Expect the budget to move with the rolling window[​](#expect-the-budget-to-move-with-the-rolling-window "Direct link to Expect the budget to move with the rolling window") SLOs evaluate over a rolling window of 7, 14, 21, 28, or 90 days. Every evaluation point answers one question: how good was the service over the previous full window from this moment? The window is always forgetting old data and adding newer data, which can look surprising when you expect the budget to move only during a new incident. **Error budget recovery** is when the remaining budget rises because old bad events or bad windows have aged out of the window. It does not mean the service improved at that moment. ### Example: old good traffic ages out[​](#example-old-good-traffic-ages-out "Direct link to Example: old good traffic ages out") This example uses a 28-day event-based SLO over a 50-day timeline. A large traffic spike of mostly-good requests arrives first, followed later by a small burst of 40 bad requests. While the spike is still inside the rolling window, the bad burst is diluted by a large denominator. Once the spike ages out, the same 40 bad requests divide into a much smaller denominator and consume far more of the budget. At day 22, the shaded window still includes the earlier traffic spike, so the 40 bad requests are diluted by a large denominator and 74.8% of the budget remains. [![Traffic chart with the 28-day window shaded over the earlier good-traffic spike; remaining budget sits at 74.8% when the bad burst lands](/docs/assets/images/denominator-effect-1-37236a7e0dad407560817c18669cc86d.webp)](https://coralogix.com/docs/docs/assets/images/denominator-effect-1-37236a7e0dad407560817c18669cc86d.webp) Shows the window still covering the traffic spike when the bad burst lands. By day 49, the traffic spike has aged out of the window, but the bad burst is still inside it. The denominator is now much smaller, so the same bad requests consume far more of the budget — 22.5% remains. [![Traffic chart with the window slid past the spike but still over the bad burst; remaining budget has dropped to 22.5%](/docs/assets/images/denominator-effect-2-831935f57842a26e28ff6e570fb833cb.webp)](https://coralogix.com/docs/docs/assets/images/denominator-effect-2-831935f57842a26e28ff6e570fb833cb.webp) Shows the spike aged out while the bad burst remains inside the window. By day 50, the bad burst ages out too. It no longer counts, so the remaining budget recovers to 100%. [![Traffic chart with the window slid past the bad burst; remaining budget recovers to 100%](/docs/assets/images/denominator-effect-3-8d4477df8e975ad1b71e56b332a225a6.webp)](https://coralogix.com/docs/docs/assets/images/denominator-effect-3-8d4477df8e975ad1b71e56b332a225a6.webp) Shows the bad burst aged out of the window. ### Example: an incident ages out and the budget recovers[​](#example-an-incident-ages-out-and-the-budget-recovers "Direct link to Example: an incident ages out and the budget recovers") This example also uses a 28-day event-based SLO over a 50-day timeline. A bad-event spike happens and stays inside the rolling window for a while. Once it falls outside the window, the SLO stops counting it and the remaining budget jumps back up — that jump is error budget recovery. At day 13, the incident spike is fully inside the current window, so it counts against the SLO and pushes the remaining budget down to 7.7%. [![Bad-events chart with the 28-day window shaded over the incident spike; remaining budget drops to 7.7%](/docs/assets/images/budget-recovery-1-37ce9396eeb7a73151efa560c1efa8f8.webp)](https://coralogix.com/docs/docs/assets/images/budget-recovery-1-37ce9396eeb7a73151efa560c1efa8f8.webp) Shows the incident spike fully inside the current window. By day 40, the incident sits at the left edge of the window. It is old, but it still counts, so the budget has only partly recovered to 78.6%. [![Bad-events chart with the incident spike at the left edge of the window; remaining budget partly recovered to 78.6%](/docs/assets/images/budget-recovery-2-addc8d6c05c25207d50c1088427562b7.webp)](https://coralogix.com/docs/docs/assets/images/budget-recovery-2-addc8d6c05c25207d50c1088427562b7.webp) Shows the incident spike at the left edge of the window. By day 41, the incident has moved outside the window. It no longer counts, so the remaining budget jumps back to 100%. That jump is error budget recovery. [![Bad-events chart with the incident spike outside the window; remaining budget jumps back to 100%](/docs/assets/images/budget-recovery-3-95f0b9a2a04cb403153f8204e3105db0.webp)](https://coralogix.com/docs/docs/assets/images/budget-recovery-3-95f0b9a2a04cb403153f8204e3105db0.webp) Shows the incident spike aged out of the window. ### Time-window SLOs count a bounded set of windows[​](#time-window-slos-count-a-bounded-set-of-windows "Direct link to Time-window SLOs count a bounded set of windows") Time-window SLOs count windows rather than individual requests, so the rolling window holds a fixed number of slots. With 5-minute windows, a 7-day SLO has 7 × 24 × 12 = 2,016 windows and a 28-day SLO has 28 × 24 × 12 = 8,064. The query, operator, and threshold mark each window good or bad. Because a busy 5-minute window and a quiet one each count once, a traffic spike adds no extra weight. This usually makes the denominator effect smaller and easier to reason about than an event-based SLO, where a spike can add thousands of events to the denominator. Time-window SLOs can still move sharply when: * A cluster of bad windows enters the rolling window. * A cluster of bad windows leaves the rolling window. * Missing-data handling changes which windows are counted. * Very few windows are evaluated because the service is sparse. ### What to check when the budget moves unexpectedly[​](#what-to-check-when-the-budget-moves-unexpectedly "Direct link to What to check when the budget moves unexpectedly") When the budget moves and no new incident happened, check the inputs before assuming the SLO is wrong: * For event-based SLOs, did the denominator change because old good traffic entered or left the window? * For event-based SLOs, did an old incident finally age out? * For time-window SLOs, did a cluster of good or bad windows enter or leave the window? * Did grouping split the traffic so much that one small burst now has a large effect? * Did missing data or ingestion delays change which events or windows were counted? Pair SLOs with direct alerts for real-time detection. An SLO explains reliability over the window; it does not always explain what is broken right now. ## Build the query before you choose the target[​](#build-the-query-before-you-choose-the-target "Direct link to Build the query before you choose the target") A good SLO starts with a good query, not with a target. Work through these steps in order. 1. **Define the user-impact question.** Good questions read like "what percentage of checkout requests completed below 500 ms?" If the question is "is it broken right now?", use a direct alert instead. 2. **Choose the SLO type** using the table in [Match the SLO type to the signal](#match-the-slo-type-to-the-signal). 3. **Build the total query first.** Total is the population you are judging — all customer-facing requests, all checkout requests, all runs of a job. Exclude health checks, internal retries, synthetic load, and traffic outside the reliability promise. 4. **Build the good query as a subset of total.** Good must be the same population plus the success condition. Comparing good `/api` requests against total requests for every route understates compliance and is hard to explain. 5. **Use `increase()` for event-based counts, not `rate()`.** `rate(counter[1m])` returns events per second, which is a throughput signal rather than a count. For event-based SLOs, `good` and `total` should be event counts so `52 bad requests out of 10,000` stays meaningful. Reserve `rate()` for dashboards, throughput alerts, or time-window SLOs that explicitly evaluate a per-second rate. 6. **Match the grouping.** If good groups `by (service, route)`, total must group `by (service, route)` too. Mismatched grouping produces one series per service+route on one side and one series per service on the other. 7. **Check the query shape** with the list below. 8. **Validate volume before the target.** Confirm the data can support the target instead of choosing 99.9% and hoping. A well-formed event-based pair looks like this: ``` # Good: non-5xx checkout requests sum(increase(http_requests_total{service="checkout", route!="/healthz", status!~"5.."}[1m])) by (service) # Total: all checkout requests in the same population sum(increase(http_requests_total{service="checkout", route!="/healthz"}[1m])) by (service) ``` Before creating the SLO, confirm: * Good is a subset of total. * Good and total use the same filters except for the success condition. * Good and total use the same grouping labels. * Event-based queries use a `[1m]` range vector. * Values cannot go negative, and the query aggregates rather than returning too many series. * Derived metrics from Recording Rules or Events2Metrics have enough offset or delay. * Preview values make sense, and good never exceeds total. For more query patterns, see [Query examples](https://coralogix.com/docs/docs/user-guides/slos/create/.md#query-examples) and [Preview and validate SLOs](https://coralogix.com/docs/docs/user-guides/slos/create/preview/.md). ## Design event-based SLOs[​](#design-event-based-slos "Direct link to Design event-based SLOs") Event-based SLOs are usually the safest choice for request success and error-rate objectives. Use them when you can count good and total events, the success condition is request-level, volume per permutation is high enough, and you want reliability accounting over the SLO window. Be careful when the metric is sparse, the scrape interval is close to the 1-minute range, the source has gaps, grouping splits the data too much, or the service is intermittent. An availability SLO counts non-5xx responses against all responses: ``` # Good: non-5xx requests sum(increase(http_requests_total{service="checkout", status!~"5.."}[1m])) by (service) # Total: all requests in the same population sum(increase(http_requests_total{service="checkout"}[1m])) by (service) ``` For latency, an event-based SLO built on histogram buckets is often more stable than a per-window percentile. Counting requests under a threshold answers "what percentage of requests completed within 300 ms?" rather than "was the p99 below 300 ms in this one window?": ``` # Good: requests <= 300ms sum(increase(http_request_duration_seconds_bucket{service="checkout", le="0.3"}[1m])) by (service) # Total: all observed requests sum(increase(http_request_duration_seconds_count{service="checkout"}[1m])) by (service) ``` ## Design time-window SLOs[​](#design-time-window-slos "Direct link to Design time-window SLOs") Time-window SLOs evaluate a condition for each 1- or 5-minute window and mark each window good or bad. They fit sustained-behavior objectives — p95 latency staying below a threshold in most windows, error rate staying low, queue depth or saturation not remaining high for too long. They are risky for very sparse traffic, p99 latency on low request counts, 1-minute windows on bursty traffic, or missing-data handling that isn't thought through. Time controls the denominator, so a traffic spike does not add SLO weight. A busy 5-minute window and a quiet one each count as exactly one window. [![Bar chart of request volume per 5-minute window with a spike, alongside a flat line showing each window carries one unit of SLO weight](/docs/assets/images/time-window-weighting-5be4a690caba13e2c6e83e54641a5ee9.webp)](https://coralogix.com/docs/docs/assets/images/time-window-weighting-5be4a690caba13e2c6e83e54641a5ee9.webp) Shows that a busy window and a quiet window carry equal SLO weight, which reduces the traffic-spike denominator effect that event-based SLOs can have. Two design rules keep time-window SLOs stable: * **Prefer 5-minute windows for sparse data.** A 1-minute window can be too noisy; 5 minutes gives more samples per decision. * **Give percentiles enough samples.** A p99 over three requests is essentially "one of those three requests." For sparse latency, prefer average, p90, or p95; use 5-minute windows; base the threshold on observed data; or model latency as event-based requests-under-threshold when histogram buckets exist. Decide what missing data means for each SLO — do not leave it as an afterthought: | Missing data counts as | When it fits | Risk | | ---------------------- | --------------------------------------------------------------------------- | ------------------------------------------------ | | Uncounted | Bursty services where no traffic should neither help nor hurt | Few windows scored, increasing variance | | Good | A metric that only emits during failures, or where no traffic is acceptable | Can hide real outages if absence is actually bad | | Bad | Heartbeats or jobs that must report continuously | Can punish normal idle periods | ## Handle no-data, idle, and intermittent services[​](#handle-no-data-idle-and-intermittent-services "Direct link to Handle no-data, idle, and intermittent services") No data is not the same as healthy. A request-ratio SLO can only evaluate requests that exist — if a service receives no requests, there are no bad requests to count, but that proves nothing about availability. Match the signal to the case: | Case | Better signal | | ----------------------------------------- | ------------------------------------------------------------------- | | Service must receive regular traffic | Alert when request volume drops below the expected baseline | | Heartbeat endpoint must be called | Absence or no-data alert | | Batch job must run hourly or daily | Alert when there is no successful run by the expected time | | Service has intermittent customer traffic | SLO for reporting, direct alerts for hard failures | | SLO query has gaps | Fix instrumentation, or avoid SLO-based paging for that permutation | For intermittent permutations, budget movement can be misleading, because the SLO lacks enough continuous data to produce a stable signal. ## Choose the right alert[​](#choose-the-right-alert "Direct link to Choose the right alert") A healthy reliability setup usually pairs SLOs for accountability with direct alerts for immediate detection. Coralogix offers two [SLO alert](https://coralogix.com/docs/docs/user-guides/slos/alerts/.md) conditions plus standard alerts: | Alert | Answers | Best for | Avoid when | | ---------------------------------- | ----------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------ | | **Error budget** | How much budget is left? | Slow-moving reliability risk; warning before breach | The SLO is new, sparse, or not trusted yet | | **Burn rate** | How fast is the budget being spent? | High-volume SLOs where a sharp drop signals real degradation | Low-volume, bursty, sparse, or noisy SLOs | | Direct metric, log, or trace alert | Is something broken right now? | Immediate action — high 5xx rate, high latency, failed job, missing heartbeat, saturation | Long-term reliability reporting | An [Error budget alert](https://coralogix.com/docs/docs/user-guides/slos/alerts/.md#error-budget-alert) is the safest SLO alert for low- and medium-volume SLOs, because it does not assume short-term movement is always meaningful. A [Burn rate alert](https://coralogix.com/docs/docs/user-guides/slos/alerts/.md#burn-rate-alert) catches a sharp drop over a lookback, so use it only when the SLO has enough volume for the drop to mean something. Before enabling one, inspect historical behavior and ask whether it would have fired only on incidents you actually wanted to act on. Use this matrix to avoid turning every reliability question into the same alert: | Need | Best answer | | ---------------------------------------- | -------------------------------------------------------------- | | Measure long-term reliability | Create an SLO | | Alert when budget is nearly gone | Error budget alert | | Alert when budget drops sharply | Burn rate alert, for high-volume, stable SLOs | | Page on a hard outage | Direct metric, log, or trace alert | | Page on one critical job failure | Direct job-failure alert; optionally keep an SLO for reporting | | Detect no traffic or a missing heartbeat | No-data or absence alert | | Report reliability to customers or teams | Create an SLO and report compliance and error budget | Do not page from a low-volume SLO just because it is an SLO. If one event can dominate the budget, keep the SLO for reporting and page from a direct, concrete failure signal instead. ## Let the SLO warm up before you trust it[​](#let-the-slo-warm-up-before-you-trust-it "Direct link to Let the SLO warm up before you trust it") Data collection for an SLO's metric begins at creation, and initial calculations cover only data from that point onward. Once the SLO is older than its time frame, it evaluates a full rolling window. So a new SLO is not a reliable 7/14/21/28/90-day statement until one complete window has elapsed, and editing an SLO can reset historical evaluation for the previous configuration. We recommend this sequence: 1. Preview and validate the SLO. 2. Create it without paging alerts, unless the signal is already proven. 3. Let it run for a full window. 4. Review volume, bad events, grouping, and budget behavior. 5. Add alerts once the SLO behaves as expected. ## Pre-flight checklist[​](#pre-flight-checklist "Direct link to Pre-flight checklist") Run through these checks before creating an SLO. ### SLO type[​](#slo-type "Direct link to SLO type") * For an APM service, an APM SLO was considered before custom PromQL. * Request success or error rate is modeled as event-based where possible. * Latency is modeled as either request-level success rate or pass/fail windows, deliberately. * For batch, heartbeat, or no-traffic cases, an SLO is actually the right tool. ### Query correctness[​](#query-correctness "Direct link to Query correctness") * Good events are a subset of total events. * Good and total use the same filters except for the success condition. * Good and total use the same grouping labels. * Event-based queries use the required `[1m]` range. * Event-based counter SLOs use `increase(counter[1m])`, not `rate(counter[1m])`. * Values cannot go negative, and the query aggregates. * Derived metrics have enough offset or delay. * Preview shows the expected good/total or metric/status values, and good never exceeds total. ### Volume and target[​](#volume-and-target "Direct link to Volume and target") * Total events or windows are high enough for the chosen target. * The volume check was done per permutation. * The target is the loosest target that is still meaningful. * A single bad event will not unexpectedly breach the SLO, unless that is intentional. ### Grouping and ownership[​](#grouping-and-ownership "Direct link to Grouping and ownership") * Grouping labels are bounded and low-cardinality. * Expected permutations are well under the limit. * High-value tenants or routes are split into separate filtered SLOs if needed. * Ownership tags give each permutation a clear Service, Environment, and Team. ### Window and warm-up[​](#window-and-warm-up "Direct link to Window and warm-up") * The window is one of 7, 14, 21, 28, or 90 days and long enough for the traffic pattern. * Low-volume services use the longest acceptable window. * The team knows the SLO is not trustworthy until one full window has elapsed, and that edits reset historical comparisons. ### Alerting[​](#alerting "Direct link to Alerting") * Error budget alerts cover low-volume or slow-moving risk. * Burn rate alerts are used only for stable, high-volume SLOs. * Direct alerts cover hard failures, absence, job failures, and urgent conditions. * Alerts were tested against historical behavior before paging. ## Troubleshoot unexpected SLO behavior[​](#troubleshoot-unexpected-slo-behavior "Direct link to Troubleshoot unexpected SLO behavior") | Symptom | Likely cause | Fix | | ------------------------------------------------------ | ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | One error breached the SLO | Too little data for the target | Loosen the target, lengthen the window, reduce grouping, or use direct alerting | | SLO looks fine at service level but one route breached | Grouping split the traffic | Remove grouping, group by endpoint class, or create filtered SLOs for important routes | | Green summary but no useful drilldown | No data or an idle permutation | Add a traffic or absence alert if no data should count as bad | | Compliance keeps changing after creation | The SLO window is still warming up | Wait one full window | | SLO reset after an edit | New configuration | Expected — avoid editing while evaluating | | Error budget dropped with no new incident | Rolling-window denominator changed, or old good traffic aged out | Explain the rolling-window effect; lengthen the window if needed | | Error budget suddenly recovered | An old incident or bad windows aged out of the window | Expected rolling-window recovery | | Burn rate alert flaps | Low volume, sparse data, or too short a lookback | Use an Error budget alert or a direct alert | | p99 latency SLO jumps around | Too few requests per window | Use p95 or average, 5-minute windows, an event-based histogram SLO, or a direct alert | | Good events exceed total events | The good query is not a subset of total | Align the good and total query logic | | Cardinality or permutation error | Too many groups | Reduce grouping, filter, or split SLOs | | Query passes elsewhere but SLO creation fails | Coralogix SLO query restrictions | Adjust the query to an SLO-compatible shape — see [Query validation](https://coralogix.com/docs/docs/user-guides/slos/create/query-validation/.md) | | SLO says "requests" but uses `rate()` | The query returns a per-second rate, not a count | Use `increase(counter[1m])` for event-based request-count SLOs | When an SLO behaves unexpectedly, start from this: the SLO is probably measuring a signal that is too sparse, too strictly targeted, too heavily grouped, or not suited to SLO-based alerting. The fix is usually one of — loosen the target, lengthen the window, reduce grouping, switch SLO type, fix missing or inconsistent data, use a direct alert, separate high-volume services from intermittent ones, or switch event-based queries to count semantics. A good SLO is boring most of the time; when it surprises people, treat that as feedback on its design. ## Recommended defaults[​](#recommended-defaults "Direct link to Recommended defaults") For most customer-facing services: | Setting | Default | | ----------- | ---------------------------------------------------------------------------------- | | SLO type | APM or event-based | | Target | 99% initially; tighten only after observing real data | | Window | 28 days for reporting; shorter windows only when justified | | Grouping | Service, environment, region, or a controlled route class | | Query shape | `increase(counter[1m])` for event-based counts | | Alerting | Error budget plus direct alerts; Burn rate alerts only when high-volume and stable | | Latency | p95 or request-under-threshold; avoid p99 unless high-volume | | Warm-up | Wait one full window before trusting or reporting | For low-traffic services, prefer event-based SLOs if possible, otherwise keep them reporting-only. Use a 95% or 99% target, the longest acceptable window, minimal or no grouping, and average or p95 latency. Avoid Burn rate alerts, use direct alerts for immediate action, and use an absence alert when traffic is expected. For batch jobs, page from direct signals — job failed, no successful run by the expected time, or duration exceeded — and keep any SLO for reporting only, such as the percentage of scheduled runs that succeeded. Avoid sparse time-window latency SLOs and Burn rate paging. ## Related resources[​](#related-resources "Direct link to Related resources") * [Create Service Level Objectives](https://coralogix.com/docs/docs/user-guides/slos/create/.md) * [Preview and validate SLOs](https://coralogix.com/docs/docs/user-guides/slos/create/preview/.md) * [SLO alerts](https://coralogix.com/docs/docs/user-guides/slos/alerts/.md) * [Using the underlying SLO metrics](https://coralogix.com/docs/docs/user-guides/slos/underlying-metrics/.md)