Alerting & Escalation¶

The complete, top-to-bottom path through b'nerd alerting: wire Prometheus Alertmanager in, define who gets notified and how escalation pages people, layer on schedules and noise control once it's live, exercise it with the self-serve tools, set up phone/SMS delivery, and — at the end — understand exactly what happens to an alert from the moment it fires to the moment it's resolved. See the Alerting & Escalation API reference and the Telephony & SMS API reference for full request/response shapes; this guide cross-links into both rather than repeating them.

The journey:

1. Enable the capability → wire Alertmanager → define routes → build an escalation chain
2. Add schedules, overrides & absence to the roster you paged in step 4
3. Add external phone contacts for people with no b'nerd account
4. Tame the noise once real traffic starts arriving
5. Fan out to chat/webhooks
6. Try the self-serve tools — test alerts, route preview, cloning, resolved targets, insights
7. Set up phone & SMS delivery
8. Understand the alert lifecycle end to end

1. Enable the alerts capability¶

Alerting is a project capability, requested and approved the same way as any other feature — see Requesting Feature Access for the full request/approval lifecycle, dashboard flow, and auto-approve option. The short version:

POST /projects/{project_id}/feature_requests
Authorization: Bearer <token>
Content-Type: application/json

{ "feature_key": "alerts" }

Until this is approved (or auto-approved, if your org admin has enabled that for alerts), every alerting endpoint — including the ingest webhook — responds 403 alerts_capability_not_enabled.

2. Mint an ingest token and wire Alertmanager¶

Mint a project-bound, alerts:write-only token:

POST /projects/{project_id}/alert_ingest_tokens
Authorization: Bearer <token>

The response includes a ready-to-paste webhook_config snippet and the raw token value — shown once, here only:

receivers:
  - name: bnerd-alerting
    webhook_configs:
      - url: 'https://api.bnerd.cloud/projects/<project_id>/alerts'
        http_config:
          bearer_token: '<the minted token>'

Add that receiver to your Alertmanager configuration and route your alerting rules to it. Alertmanager will now POST every firing/resolved evaluation cycle to b'nerd. This token can be revoked like any other static token via DELETE /tokens/{id} if it's ever compromised — mint a fresh one and update Alertmanager's config.

3. Define alert routes¶

An alert route decides, per firing alert, who gets notified in-app and/or whether an escalation chain opens. Routes are evaluated in priority order (ascending) — the first matching route wins.

POST /projects/{project_id}/alert_routes
Authorization: Bearer <token>
Content-Type: application/json

{
  "name": "Critical — page on-call",
  "priority": 10,
  "matchers": [{ "label": "severity", "op": "eq", "value": "critical" }],
  "min_severity_for_notification": "critical",
  "in_app_target_type": "org_admins",
  "escalation_chain_id": "<chain-uuid, created in the next step>",
  "enabled": true
}

A common pattern is a small ladder of routes: a high-priority rule that pages critical alerts through an escalation chain, and a low-priority catch-all (no escalation_chain_id, in_app_target_type: "org_admins") that just notifies for everything else. Matchers within one route are AND-combined — add a catch-all route with no matchers at the bottom of your priority list if you want every remaining alert to at least raise an in-app notification.

4. Build an escalation chain¶

An escalation chain is an ordered list of steps plus the re-escalation/repeat behavior wrapped around them. Start with the roster the chain will page:

POST /projects/{project_id}/on_call_rosters
{ "name": "Platform on-call" }

POST /projects/{project_id}/on_call_rosters/{roster_id}/on_call_roster_members
{ "account_id": "<a project member's account id>" }

Then the chain itself:

POST /projects/{project_id}/escalation_chains
{
  "name": "Production on-call",
  "enabled": true,
  "re_escalation_after": 1800,
  "repeat_mode": "n_times",
  "max_repeats": 3,
  "repeat_delay_seconds": 300
}

And its steps:

POST /projects/{project_id}/escalation_chains/{chain_id}/escalation_steps
{ "step": { "position": 1, "target_type": "roster", "target_roster_id": "<roster_id>",
            "channels": ["in_app", "email"], "delay_seconds": 0,
            "ack_timeout_seconds": 300, "repeat": 1 } }

POST /projects/{project_id}/escalation_chains/{chain_id}/escalation_steps
{ "step": { "position": 2, "target_type": "org_admins",
            "channels": ["in_app", "email"], "delay_seconds": 0,
            "ack_timeout_seconds": 600, "repeat": 1 } }

A step's target_type is roster (page every roster member), member (page one specific account), or org_admins (page every organization admin) — no roster required for that last one. Finally, point the route you created in step 3 at this chain's escalation_chain_id.

5. Add schedules to a roster (optional)¶

A roster's static membership list from step 4 is always the ultimate fallback, but it can layer time-based scheduling on top: one rotation that hands off between its participants on a repeating cadence, plus any number of ad-hoc overrides for one-off coverage swaps. Both are entirely optional — a roster with neither behaves exactly as before this feature existed.

POST /projects/{project_id}/on_call_rosters/{roster_id}/rotation
{
  "rotation_type": "weekly",
  "handoff_time": "09:00",
  "handoff_weekday": 1,
  "timezone": "Europe/Berlin",
  "anchor_date": "2026-07-06",
  "enabled": true,
  "participants": [
    { "account_id": "<account-a>", "position": 0 },
    { "account_id": "<account-b>", "position": 1 }
  ]
}

Handoffs land at handoff_time local wall-clock time in timezone, and stay there across a daylight-saving transition instead of drifting by an hour — see the DST note in the API reference for exactly how a handoff inside a spring-forward gap resolves.

Cover a one-off swap without touching the standing rotation:

POST /projects/{project_id}/on_call_rosters/{roster_id}/on_call_overrides
{
  "account_id": "<covering-account>",
  "starts_at": "2026-07-10T00:00:00Z",
  "ends_at": "2026-07-11T00:00:00Z",
  "reason": "Swap with Bernd"
}

While an override is active it entirely replaces the rotation's computed participant for that window (two simultaneous overrides both apply — see Resolution rules). Outside any override window, the enabled rotation's scheduled participant pages instead.

See who's on call right now, or preview upcoming handoffs, with GET .../shifts — the same server-side computation the paging engine itself uses, so it can't disagree with reality.

Marking yourself absent (vacation mode)¶

Absence doesn't suppress paging — it changes who gets paged (and who gets skipped) while an account is away. Any account can record its own absence windows via Account → Notification Preferences → Absence, or the API:

POST /notification_absences
Authorization: Bearer <token>
Content-Type: application/json

{ "starts_at": "2026-07-14T00:00:00Z", "ends_at": "2026-07-21T00:00:00Z", "note": "Summer vacation" }

While an absence is active, the account is excluded from whatever this roster's resolution rules would otherwise resolve to — before the "never page nobody" fallback above even runs. On the roster page, each member/participant row shows a yellow away badge, and a roster whose entire resolved on-call is currently absent shows a warning banner — "Everyone on call is absent — alerts will still page them." — with a shortcut to add an override.

See Absence exclusion for the exact resolution order and response fields, and the Absence / vacation mode API reference for the full request/response shape.

6. Add external phone contacts to a roster (optional)¶

A roster can also page people with no b'nerd account at all — an external NOC hotline, a vendor's own on-call line — as an external phone contact. A contact only ever receives phone- or sms-channel escalation (it has no account to notify in_app or by email); add one alongside step 4's account members:

POST /projects/{project_id}/on_call_rosters/{roster_id}/on_call_roster_phone_contacts
{
  "label": "Vendor NOC (Acme)",
  "phone": "+493012345678",
  "consent": true
}

consent must be truthy — omitting or falsifying it fails with 422 consent_required, never a silent no-op. The platform then records who added the contact and when consent was acknowledged instead of placing a verification call, which is why a contact is exempt from the verified-phone gate that otherwise applies to account targets — see DSGVO: consent replaces verification.

A contact acknowledges an escalation call exactly like an account does — pressing 1 on the call. Because it has no account, its acknowledgements and timeline entries are attributed by the label you gave it rather than an account { id, name }.

7. Tame the noise¶

Once alerting is live, the next problem is usually too much paging, not too little: a flapping alert refiring every minute, a planned deploy that's expected to trip a dozen rules, or 2am email for something that can wait until morning. b'nerd gives you five independent tools for this — reach for the narrowest one that fits:

Silence is covered in The alert lifecycle below (it's a per-alert action you take on something already firing); absence is covered in step 5 above (it's an on-call concern, not a route/project setting). The remaining three live here.

Per-route cooldown¶

An alert route can carry a notification_cooldown_seconds field (integer, >= 0, default 0 — no cooldown). While a route is within its cooldown window, a matching alert's recurrence still gets routed and its timeline still records the match, but the route's notify (in-app/email) and notification endpoint fan-out are suppressed — recorded as a route_suppressed timeline event with reason: "cooldown". The cooldown clock resets from each genuine notify, so a flapping alert paging every minute instead pages at most once per cooldown window.

PATCH /projects/{project_id}/alert_routes/{id}
Authorization: Bearer <token>
Content-Type: application/json

{ "notification_cooldown_seconds": 900 }

Cooldown has no dashboard field yet — set it via the API on the route you want throttled. Good candidates: an endpoint-only route (chat/webhook fan-out with no escalation) fielding a chronically flapping check, or an in-app-only catch-all that would otherwise spam the inbox on every recurrence.

Maintenance windows¶

A maintenance window is the scheduled counterpart to a silence: instead of snoozing one already-firing alert, you schedule a suppression window ahead of time for whatever matches a set of label matchers — the same eq/re, AND-combined language alert routes use. While a window is active, it suppresses everything for matching alerts: in-app/email notify, escalation (no instance opens or re-opens), and endpoint fan-out on both firing and resolved deliveries. It's checked before the silence check, before cooldown, before routing even resolves a chain — nothing else in the noise-control stack outranks it.

POST /projects/{project_id}/maintenance_windows
Authorization: Bearer <token>
Content-Type: application/json

{
  "name": "DB migration — 2026-07-10",
  "matchers": [{ "label": "service", "op": "re", "value": "^db-.*" }],
  "starts_at": "2026-07-10T22:00:00Z",
  "ends_at": "2026-07-10T23:30:00Z",
  "note": "Planned schema migration, expect noisy replication-lag alerts"
}

An empty (or omitted) matchers array is a project-wide catch-all — useful for "everything's expected to be noisy during this window" rather than targeting specific services. A window is active on [starts_at, ends_at) and, per v1, is one-shot — recurring maintenance (every weekend, every deploy) needs a fresh window each time, either by hand or from your own deploy automation calling the API.

Dashboard: Project → Alerts → Maintenance lists windows partitioned into Active, Upcoming, and Past, and lets you create, edit, or cancel one from there.

Quiet hours and email digests¶

This is a per-account schedule, not a per-alert or per-route setting — it governs when your email arrives, across every project and org you belong to. Configure it under Account → Notification Preferences → Email delivery, or via the API — see the Email delivery schedule API reference for the full request/response shape and field validation.

Three modes:

• Instant (default) — every email sends right away.
• Hourly digest — non-mandatory email accumulates and is batched into roughly one summary email per hour.
• Daily digest — non-mandatory email accumulates all day and flushes in one summary email at a configured local hour.

Layered on top, an optional quiet hours window ("HH:MM"–"HH:MM", wraps midnight, evaluated in your chosen time zone) holds non-mandatory email regardless of mode — even in instant mode, nothing sends while you're in your quiet window. Held items flush as soon as quiet hours end, or at the next digest cadence boundary, whichever applies. SMS has no digest equivalent — a non-mandatory SMS during quiet hours is skipped outright, not held; see SMS notification channel.

Nothing is ever silently dropped: a held item becomes a NotificationDigestItem row the moment it would otherwise have sent, and the recurring DigestFlushJob (below) is what turns accumulated items into the one summary email you actually see.

Recurring jobs (operations)¶

Two Solid Queue recurring jobs back this section's email-scheduling and retention behavior — relevant for capacity-planning the worker deployment, not something you interact with directly:

If you self-host webhook receivers on a private network and rely on ALLOW_PRIVATE_WEBHOOKS for that deployment, note that it's a platform-wide, trusted/dev-only escape hatch — not something to enable on a production deployment accepting endpoint URLs from outside full operator trust.

8. Fan out to Slack, Teams, or your own webhook (optional)¶

Beyond paging people, a route can fan an alert out to a machine or a chat tool — a notification endpoint. Create one, then attach it by id to a route's notification_endpoint_ids:

POST /projects/{project_id}/notification_endpoints
{
  "name": "Ops automation bridge",
  "kind": "webhook",
  "url": "https://hooks.example.com/bnerd-alerts"
}

The 201 response includes a secret field — the raw HMAC signing key, shown once, here only. Save it now; there is no "reveal again" endpoint. Then attach it to a route from step 3:

PATCH /projects/{project_id}/alert_routes/{route_id}
{ "notification_endpoint_ids": ["<the endpoint id above>"] }

Attaching replaces the route's full endpoint set — send every id you want attached, not just the new one; [] detaches all. From that point on, every alert this route matches delivers to the endpoint on both firing and resolved. Send a test through the real delivery pipeline before wiring up a production receiver:

POST /projects/{project_id}/notification_endpoints/{endpoint_id}/test

The same notification_endpoint_ids field also exists on an escalation step — attach it there instead of (or alongside) a route when you want the endpoint to fire time-staged with the chain rather than immediately on every match. See the step 4 tip above and Attaching endpoints to escalation steps for the full semantics.

Try it: self-serve conveniences¶

Once routes and chains exist, five self-serve tools (Wave 7) let you exercise and verify your setup without waiting for a real Alertmanager delivery. All five are also on the dashboard — Project → Alerts and its Routes/ Escalation Chains tabs.

Try it: send a test alert¶

The Alerts tab's header has a Send test alert button; each row on the Routes tab has its own per-route Test button. Both call POST .../alerts/test — a real alert, routed through the real engine, not a simulation.

A fired test alert carries a test badge and a Resolve now button (on the Alerts tab) so you don't have to wait out the 10-minute auto-resolve — see Send a test alert for what "resolve" actually does under the hood (full resolved-transition parity with a real alert: timeline event, escalation cancel, endpoint fan-out).

Preview a route before you save it¶

The Routes tab's Preview matching panel (above the route list) lets you enter hypothetical labels and see, before you ever create or edit a route, which route would match, who'd be notified, whether it'd escalate, which endpoints would fan out, and whether an active maintenance window would suppress it — a dry run against POST .../alert_routes/preview. Nothing is created; it's safe to experiment freely, including a deliberately non-matching label set to confirm your catch-all route's priority ordering is what you expect.

Because it evaluates a hypothetical alert that doesn't exist yet, preview can only report maintenance-window suppression — a real silence or cooldown state is tied to an actual alert's own history, which a preview candidate doesn't have. To check whether a specific real alert is silenced or in cooldown, look at that alert directly instead.

Cloning routes & chains¶

Both the Routes and Escalation Chains tabs have a Clone button per row — the fastest way to build a family of near-identical routes/chains (e.g. one per service, same escalation shape, different label matchers). The clone is named "Copy of {name}", created disabled, and appears at the back of the list (a cloned route sorts to the lowest priority so it can't accidentally steal traffic from its source before you've reviewed it) — the dashboard scrolls it into view and briefly highlights it so you don't have to hunt for what you just created. Review and edit the clone (matchers, targets, priority), then enable it.

See Clone a route / Clone a chain for exactly what gets duplicated — including the note that a cloned step still targets the same roster/member as its source until you point it elsewhere.

Who's actually on call right now (resolved targets)¶

The Escalation Chains tab's "Who gets paged now?" toggle, per chain, expands an inline panel showing each step's target type and the actual account names GET .../resolved_targets resolves right now — rotation handoffs, active overrides, and absences all already applied, since it's a pure read against the identical resolution method real paging uses. A step showing no resolved accounts ("nobody currently resolves") is a legitimate, honest result — an empty roster with no rotation and no members — not an error. This is a pure read; expanding it fires no alert and pages nobody.

Starting from recommended defaults¶

A brand-new project with the alerts capability enabled but nothing configured shows a Seed recommended defaults action on the Routes tab's empty state, alongside Add a route manually. It wraps POST .../alerts/seed_presets to create a starter set of routes and escalation chains in one call — a faster starting point than building steps 3–4 above from scratch, if the defaults are close enough to what you want.

The CTA only appears when the project is genuinely empty — zero routes and zero escalation chains. If a chain already exists but no route does (or vice versa), seeding would 422 (presets_already_configured), so the dashboard doesn't offer it in that state — only the manual "Add a route"/"New chain" path is shown, since seeding wouldn't help.

Searching and paginating the alert list¶

The Alerts tab has a search box (matches alertname, fingerprint, or the summary annotation) and Firing/Resolved/Acknowledged filter chips, both driving the same server-side q/status/acknowledged query params that GET .../alerts itself accepts — not a client-side filter over an already-fetched list. Results are paginated; the footer shows "Showing X to Y of Z results" using the response's real total count across every matching alert, not just the current page.

Insights: turning history into action¶

The Insights tab (in the same tab strip as Routes/Escalation Chains/ On-Call Roster) turns a project's alert history into six at-a-glance metrics — a Last 7/30/90 days window picker at the top drives all of them at once, backed by GET .../alerts/insights. Unlike the alert list's limit/offset (which silently clamp), the window picker only ever offers the three valid values — there's no invalid state to hit from this UI.

Stat cards — Alerts, Firing now, Acked share, and MTTA (median / p90) with a "Median time to first ack" subtitle. MTTA is exactly what it says: one sample per alert, timed from that alert firing to its first acknowledgement — an alert three different people all acknowledged still contributes only one data point, not three, so a heavily-visible alert doesn't skew the median just because more people happened to click ack on it. An em dash (—) means no alert in the window has been acknowledged yet, not a zero.

Volume — a stacked bar chart, one bar per day, segmented by severity. The day axis is labeled in UTC calendar days — the same boundary the server buckets by. Watch for the case shortly after your own local midnight: an alert that fires at 00:30 your time (say, UTC+2) can still land under what looks like "yesterday," because in UTC it's 22:30 the day before. This isn't a display bug; it's the one unambiguous boundary a project shared across contributors in different timezones can use.

Noisiest alerts — the same alerts (by alertname) firing and refiring over and over, ranked by occurrence count. Treat a name sitting persistently at the top of this table as a signal to act, not just observe: give its route a cooldown if it's a flapping check nobody's fixed yet, schedule a maintenance window around known-noisy planned work, or tighten the route's matchers/silence it if it's genuinely not actionable. A name that never leaves this table across multiple time windows is worth escalating as a real reliability problem, not just a paging-noise one.

On-call load — per account, how many times they were notified, called, and acked an alert in the window. Use this as a fairness check against the rotation you set up earlier: if one participant's called/notified counts are consistently far above their rotation peers', check whether the rotation's handoffs are actually landing where you expect, or whether an override or an unusually noisy period skewed one person's shift. Rows reflect historical ledger data — who was actually paged at the time — so a former team member who's since left the roster can still appear here for a window that includes their old shifts.

Telephony & SMS setup¶

Phone calls and SMS (Wave 9) share one prerequisite: a verified phone number on the account you want reached. This is the account-level setup that makes the phone/sms notification channels and the phone/sms escalation step channels above actually deliver something, rather than silently skipping every target.

Register and verify a phone number¶

Go to Account → Security → Phone Number, or via the API:

POST /accounts/me/phone
Authorization: Bearer <jwt>
Content-Type: application/json

{ "phone": "+4915112345678", "delivery_method": "sms" }

delivery_method defaults to "sms" (Wave 9) — a text with a 6-digit code. Pass "voice" for the pre-existing automated call instead. Either way, confirm with the code:

POST /accounts/me/phone/verify
Authorization: Bearer <jwt>
Content-Type: application/json

{ "code": "482913" }

See the Telephony & SMS API for the full field reference, error codes, the 4-requests-per-hour send throttle, and why the verify step itself has its own independent (and much tighter) 5-attempt limit instead.

Notification channels vs. escalation channels¶

Two independent places phone/sms show up, easy to conflate:

• Notification channels — a per-account, per-category opt-in (PATCH /notification_preferences, channel: "phone" or "sms") for any platform notification, not just alerts. Locked in the dashboard's preferences matrix until the account has a verified phone number. See Phone / SMS notification channels.
• Escalation channels — phone/sms in an escalation step's channels, above. This is a paging action, independent of the target's own notification preferences — it fires because the step says to, not because the account opted in. Only verified accounts (and, for phone/sms, consented external phone contacts) are reached either way.

Usage and cost visibility¶

Every real call/text writes an internal, org-attributed usage row — see Usage metering for what's recorded (voice seconds, SMS segments) and its honest limitations (an approximated segment count; verification voice calls whose duration can never be measured under the current architecture). There is no customer-facing usage endpoint or billing built on this yet — it exists today for internal visibility only.

Operator note: a provider must be configured¶

Both channels are safe-by-default: without a provider configured, every call/text runs on NullCaller/NullSms — logged, never actually sent. Voice is Twilio-only; SMS can come from either Twilio or sipgate (b'nerd's own production choice — no Twilio SMS number is provisioned in production). If you're a b'nerd operator standing up a new environment (not a customer configuring your own alerting), see Operator setup (voice & SMS providers) for the required environment variables, the two-provider precedence rule, and the country allowlist that gates every outbound send regardless of provider.

The alert lifecycle¶

This is the part worth understanding before you rely on it in production — the engine is deliberately conservative about never double-paging and never silently dropping an alert.

Fire → route¶

Alertmanager POSTs a firing alert. On the alert's first sighting (or a recurrence, see below), the alert is matched against the project's routes. The first matching route:

• delivers an in-app notification to the route's in_app_target_type recipient (org admins by default, or a roster/member/role — see In-app notification targets), unless it's none, if the alert's severity meets min_severity_for_notification (or the route has no minimum set — min_severity_for_notification is optional and defaults to "any severity");
• opens an escalation instance on the route's chain, if one is configured and it has at least one step — starting at step 1. A chain with zero steps is a deliberate record-only marker and never opens an instance (see Record-only chains); and
• fans the alert out to every enabled notification endpoint attached to the route — a webhook, Slack channel, or Teams channel gets the same firing event a human recipient would, independent of whether the route also has an in-app target or a chain. The same fan-out repeats on the alert's later resolved transition (see Notification endpoints for the payload shape, signing, and retry semantics).

An alert matching no route (or matching a route with in_app_target_type: "none", no chain, and no attached enabled endpoint) is recorded, but nothing pages or dispatches. A route can validly have in_app_target_type: "none" and no chain while still doing something, though — an attached enabled notification endpoint on its own satisfies the "a route must do something" guard, so a route that only fans out to a webhook (no human paging at all) is a normal, deliberate configuration, not a misconfiguration. A resolved delivery cancels any in-flight escalation instance for that alert immediately, regardless of which step it was on.

Step delays and ack-timeouts advance the chain¶

Each escalation step has a delay_seconds (how long to wait before this step fires, counted from when the escalation reaches it) and an ack_timeout_seconds (how long to wait for an ack once the step has fired before moving on). The engine:

1. Waits delay_seconds, then delivers the step's notifications on its configured channels (in_app, email, phone, and/or sms) to its resolved targets.
2. Schedules a watchdog for ack_timeout_seconds later.
3. If nobody acks within that window and the alert is still firing, the watchdog either enqueues the next step in the chain, or — if this was the last step — decides what to do based on repeat_mode (see Repeat & loop below).

If the alert resolves or gets acked before the watchdog fires, the chain halts where it is — the watchdog for a halted/resolved instance is a clean no-op when it eventually runs.

Acknowledge: a dead-man's-switch, not a stop¶

Acking an alert (in-app, the signed email link, or — for a phone-channel escalation step — pressing 1 on the phone call, see the Telephony API) halts the chain's progress: no more steps fire while the instance stays acked. But ack is not the same as resolve. If the escalation chain has a re_escalation_after window configured (in seconds) and the alert is still genuinely firing when that window elapses, the chain automatically restarts from step 1 — this is the ack-window dead-man's-switch. The intent: someone clicked "ack" but then got pulled away, or the underlying issue wasn't actually fixed — the platform doesn't let an ack silence a still-broken system forever.

A chain with re_escalation_after set to null or 0 means hold until resolve — an ack on that chain silences paging indefinitely, until the alert resolves or someone manually re-triggers.

Un-acking (see below) does not cancel or reschedule this dead-man's-switch job — it fires on its own timer regardless, which is what keeps the safety property intact even if someone un-acks and re-acks repeatedly.

Recurrence (resolved → firing again)¶

If an alert resolves and then starts firing again later (same fingerprint — think: a flapping metric), that resolved→firing transition is treated as a new occurrence, exactly like the alert's very first sighting: routes are re-evaluated and a fresh escalation instance opens from step 1. A still-firing redelivery of an alert that already has an active instance never re-opens a second instance or double-pages — only a genuine transition does.

Repeat & loop (after the last step)¶

When the last step's ack-timeout elapses with nobody having acked, the chain's repeat_mode decides what happens next:

The instance stays active through every loop-back — there is no admin notice on a loop-back itself, only on a final exhaustion (once/n_times).

Silence (snooze)¶

A silence suppresses paging for one alert without touching the state machine underneath: steps keep advancing, timing out, and looping on schedule — they just deliver no notifications while the silence is active. This matters because it means the moment the silence expires, the chain is exactly where it would have been anyway, not reset.

Silences resume automatically on expiry — no action needed — if the alert is still firing. A silence that outlives the alert (it resolved while silenced) simply expires quietly. Lifting a silence early (DELETE .../silences/{id}) stops future suppression but does not itself force an immediate re-page; if the alert is still firing, it resumes paging on its next genuine occurrence.

Un-ack (accidental-click recovery)¶

Clicked "acknowledge" by mistake? DELETE /projects/{project_id}/alerts/{id}/ack clears every acknowledgement recorded for the alert — acknowledged goes back to false immediately, and the audit trail records who un-acked and when. It is intentionally not a full undo of the escalation: it does not resurrect a halted escalation instance. If the alert is genuinely still firing, normal paging safety resumes via the ack-window dead-man's-switch on its own schedule, not synchronously at the moment of un-ack.

Resolve¶

Only a resolved delivery for the alert's fingerprint truly quiets it: it cancels every in-flight escalation instance outright (not paused — cancelled) and stops the recurring in-app "still firing" state. A resolve is the only event in the entire lifecycle that unconditionally stops paging, independent of ack state, silences, or repeat mode.

Alert detail view & timeline¶

Every alert has a detail page in the dashboard — header actions (ack/unack, silence), its labels and annotations, active silences, current escalation state, and a vertical timeline of everything that's happened to it, newest first: fired, routed, each step starting and notifying, calls placed and their status, acks, silences, and how the escalation eventually exhausted or resolved. It's backed by the read-only Alert timeline API — see that section for the full list of event kinds, their payload fields, and the 90-day retention window. Rows only ever record a genuine transition — a redelivered Alertmanager evaluation, a double ack click, or a redelivered Twilio status callback never appends a duplicate entry, so the timeline is a trustworthy audit trail of what actually happened, not of every request that came in.

Summary¶

fire ──▶ route (in-app? + open chain?)
           │
           ▼
       step 1 delay ──▶ notify ──▶ ack-timeout
                                       │
                     ┌── acked ────────┼──── not acked ──▶ next step
                     │                 │                       │
                     ▼                 │                       ▼
              re_escalation_after      │              (last step exhausted)
              dead-man's-switch        │                       │
              (still firing? ──▶ restart at step 1)   once / n_times / loop
                                                                │
                                                        exhausted (notify admins)
                                                        or loop back to step 1

silence  = suppress notifications only, state machine keeps advancing
un-ack   = clears the ack ledger, does not resume/cancel jobs
resolve  = the only unconditional stop — cancels all active instances