Skip to main content

Decide which notifications reach which destination

Your connectors send test notifications and your presets render the way you want. The open question is which notifications should reach which people, and how to say that in routers without ending up with one router per team per environment.

This page recommends five routers that cover most organizations, the order to build them in, and how to choose between a router, a routing label, and a destination named on the alert itself.

What you need

Note

A router carries at most three routing labels, and only these three: environment, service, and team. They are not free-form keys, so a router cannot match on a label such as region or tier. Each label holds a single value. On an alert definition the same keys carry the routing. prefix, as in routing.environment. If you have older alerts using routing.group, that key is a deprecated alias for routing.team and still matches the team label.

Routers to build first

Build these in order. The catch-all comes first because a notification that matches no router is dropped, so until the catch-all exists every gap in your routing is silent.

Each router below states the routing labels it carries, the rule condition, and why it has that shape. Conditions use Tera expressions that evaluate to true or false.

Catch-all fallback

Routing labels: none.

Rule condition: true

Destination: a low-traffic Slack channel or a shared mailbox that someone reviews daily.

A router with no routing labels matches every notification request, because matching requires only that every label on the router is present in the request. That makes it the safety net: anything your other routers miss lands somewhere visible instead of disappearing. Give the router a fallback connector as well, so a request that matches the router but no rule condition still gets delivered.

Review what arrives here weekly. Every recurring notification in the catch-all is a router you have not written yet.

Production paging

Routing labels: environment:production

Rule condition: alert.highestPriority == "P1"

Destination: PagerDuty, or another on-call tool.

One label and one condition. Keep the label set as small as the decision requires: a router carrying only environment:production matches every production notification regardless of team or service, which is what you want for paging. Adding team here would mean writing this router once per team.

Resist widening the condition. A paging router that also carries P2 trains people to ignore the pager.

Team channels

Routing labels: team:<team name>

Rule condition: true

Destination: that team's Slack channel.

One router per team, each carrying a single label. Because extra labels in the notification do not affect the match, this router catches the team's notifications across every environment and service without listing them.

No two routers can carry the same set of routing labels. Saving a second router with an identical label set fails and names the router that already holds it, so plan your label sets as a set of distinct filters rather than layering several routers on the same one.

This is the router that replaces per-alert destinations. Once it exists, an alert that changes hands follows its routing.team label to the new owner with no edit to the alert definition.

Case lifecycle

Routing labels: team:<team name>

Rule condition: case.priority == "P1"

Destination: the same team channel as above, using a Cases preset.

Cases inherit their routing labels from Ownership Tags on the underlying infrastructure rather than from labels you type, and each router keeps its Cases rules separate from its alert rules. That separation is the point: a team can page on P1 alerts while taking Case updates as channel messages, from one router.

Use this router for lifecycle events, not for the initial alert. The alert already reached someone through production paging or the team channel.

Non-production

Routing labels: environment:staging

Rule condition: true

Destination: a channel nobody is expected to watch in real time.

Staging and development notifications are worth keeping and not worth interrupting anyone for. Routing them explicitly also stops them filling the catch-all, which keeps the catch-all useful as a gap report.

When to use a router, a routing label, or a destination on the alert

Three surfaces can influence where a notification lands, and they are not alternatives to each other. They stack.

  • A routing label on the alert definition says who owns this signal. It is one of routing.environment, routing.service, or routing.team, and it is the only one of the three surfaces that belongs to the alert. Use it to state ownership, never to state a destination.
  • A router says where notifications for that owner go. Put every destination decision here. A destination that lives in a router can change without anyone reopening an alert.
  • A destination named on the alert definition binds one alert to one connector and preset. It is the right choice only for a genuine exception, such as a single alert that has to reach a customer-facing endpoint no router should serve.

The alert wizard reflects this: it matches notifications to routers by routing label rather than by hard-coded destination, and shows No matching routers when the labels you selected reach nothing. Treat that message as a prompt to add a label your routers already carry, not as a prompt to name a destination on the alert.

A useful test: if you can name the team but not the channel, you are looking at a routing label. If you can name the channel, you are looking at a router.

Quick reference

You want toRouter labelsConditionDestination
Page on-call for production incidentsenvironment:productionalert.highestPriority == "P1"PagerDuty
Send a team everything it ownsteam:<name>trueTeam Slack channel
Follow a Case through its lifecycleteam:<name>case.priority == "P1"Team Slack channel, Cases preset
Keep staging noise out of the wayenvironment:stagingtrueLow-attention channel
Catch what no other router matchednonetrueReviewed channel or mailbox
Route one alert to one endpointnot applicablenot applicableDestination on the alert definition

Why a notification did not arrive

Work down this list in order. The first three account for most cases.

  1. No router matched the request. A router matches only when every label on the router is present in the request. A router carrying team:sre and environment:production does not match a request that has only team:sre. Remove labels from the router to widen it.
  2. A router matched but no rule condition returned true. The request is dropped unless the router has a fallback connector. Check the condition against the request's actual priority or status.
  3. The alert carries no routing labels. An alert with no labels reaches only routers that carry no labels, which means the catch-all and nothing else.
  4. The delivery was attempted and failed. Query notification.deliveries for the failure category and the error the destination returned. Alerts record failures there; Cases record successes as well, so an absent Case record means the notification was never sent rather than sent and rejected.
Last updated on