Alerting & Escalation API¶

The alerting API ingests Alertmanager-compatible webhooks, matches firing alerts against a project's routing rules, and — optionally — pages an on-call chain until someone acknowledges or the alert resolves. All endpoints are project-scoped and require the project's alerts capability to be enabled.

See the setup & lifecycle guide for the full walkthrough (enable → mint an ingest token → wire Alertmanager → build a chain → what happens when an alert fires).

Authorization¶

All alerting resources use the shared envelope and the platform's usual authentication (session cookie, JWT Bearer, or a static access token).

Static-token scopes¶

Each controller in the alerting family is its own grantable static-token scope resource — there is no single umbrella alerts:* scope that covers routes, chains, steps, and rosters. Scope levels are read, write, manage, or * (see Authentication & Tokens).

An Alertmanager ingest token is deliberately narrow: it is a project-bound static token carrying only alerts:write, good for POST /projects/{project_id}/alerts and nothing else in this API.

RBAC (interactive sessions)¶

For session/JWT callers, fine-grained authorization runs through AlertAbility (CanCanCan). Organization admins can manage every alerting resource in the org's projects. Non-admin members need the project's alerts permission category granted at the appropriate level (read / create / update / delete) — the same permission-preset mechanism used elsewhere in the platform (see Roles & RBAC). delete is the manage-equivalent level: it is what's required to mint ingest tokens and create/delete escalation chains, routes, and rosters.

Alert ingestion¶

Ingest alerts (Alertmanager webhook)¶

POST /projects/{project_id}/alerts
Authorization: Bearer <ingest-token>
Content-Type: application/json

Point an Alertmanager webhook_config receiver directly at this URL. Accepts either Authorization: Bearer <token> or X-Access-Token: <token> — mint the token with POST /alert_ingest_tokens below, which also returns a ready-to-paste webhook_config snippet.

Alerts are upserted by fingerprint — Alertmanager redelivers the same firing alert on every evaluation cycle, so this endpoint is safe to hammer. A firing alert seen for the first time (or recurring after having resolved) is evaluated against the project's alert routes; a resolved alert cancels any in-flight escalation for it. A token scoped to a different project is rejected with 403 static_token_cross_project — the same project-envelope convention every other project-scoped consumer in this API relies on (see Error responses).

Request body — Alertmanager webhook v4 payload:

{
  "version": "4",
  "groupKey": "{}:{alertname=\"HighCPU\"}",
  "status": "firing",
  "receiver": "bnerd-alerting",
  "groupLabels": { "alertname": "HighCPU" },
  "commonLabels": { "alertname": "HighCPU", "severity": "critical" },
  "commonAnnotations": { "summary": "CPU > 90% for 5m" },
  "externalURL": "https://alertmanager.example.com",
  "alerts": [
    {
      "status": "firing",
      "labels": { "alertname": "HighCPU", "severity": "critical", "instance": "web-01" },
      "annotations": { "summary": "CPU > 90% for 5m" },
      "startsAt": "2026-06-30T20:00:00Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "fingerprint": "a1b2c3d4e5f6",
      "generatorURL": "https://prometheus.example.com/graph?g0.expr=..."
    }
  ]
}

Response 200 — always 200 for a well-formed payload; per-alert failures are reported in data.errors, not as an HTTP error, so Alertmanager doesn't retry-storm the whole batch over one bad alert. A payload that fails entirely (zero processed, at least one error) responds 502 Bad Gateway instead, so Alertmanager retries rather than silently losing the alert.

{
  "metadata": { "organization_id": "org-uuid", "project_id": "proj-uuid", "timestamp": "2026-06-30T20:00:00Z" },
  "data": {
    "processed": 1,
    "errors": []
  }
}

Response 422 — malformed payload (missing/empty alerts array):

{
  "request_id": "01970000-...",
  "errors": [
    { "code": "malformed_payload", "message": "Request body must include a non-empty `alerts` array (Alertmanager webhook v4 payload).", "status": 422 }
  ]
}

Mint an ingest token¶

POST /projects/{project_id}/alert_ingest_tokens
Authorization: Bearer <token>

Mints a project-bound StaticAccessToken carrying only alerts:write — the credential Alertmanager's webhook_config presents to the ingest endpoint above. Requires alerts:manage (see Authorization).

Response 201 — the raw token is shown once, here only; it cannot be retrieved later (list/revoke it via the generic GET/DELETE /tokens(/:id) — an ingest token is an ordinary static token owned by the issuing account):

{
  "metadata": { "organization_id": "org-uuid", "project_id": "proj-uuid", "timestamp": "2026-06-30T20:00:00Z" },
  "data": {
    "token_id": "uuid",
    "token": "bnat_...",
    "scopes": ["alerts:write"],
    "project_id": "proj-uuid",
    "webhook_config_snippet": "receivers:\n  - name: bnerd-alerting\n    webhook_configs:\n      - url: 'https://api.bnerd.cloud/projects/proj-uuid/alerts'\n        http_config:\n          bearer_token: '...'"
  }
}

Alerts¶

List alerts¶

GET /projects/{project_id}/alerts
Authorization: Bearer <token>

Returns alerts in the project, most-recently-received first (deterministic tiebreak on id descending when two alerts share a timestamp).

Query parameters (Amendment A5, Wave 7)

Response 200

{
  "metadata": { "organization_id": "org-uuid", "project_id": "proj-uuid", "timestamp": "2026-06-30T20:00:00Z", "count": 1, "total": 5 },
  "data": [
    {
      "id": "uuid",
      "project_id": "proj-uuid",
      "fingerprint": "a1b2c3d4e5f6",
      "status": "firing",
      "severity": "critical",
      "labels": { "alertname": "HighCPU", "severity": "critical", "instance": "web-01" },
      "annotations": { "summary": "CPU > 90% for 5m" },
      "starts_at": "2026-06-30T20:00:00Z",
      "ends_at": null,
      "generator_url": "https://prometheus.example.com/graph?g0.expr=...",
      "last_received_at": "2026-06-30T20:05:00Z",
      "created_at": "2026-06-30T20:00:00Z",
      "updated_at": "2026-06-30T20:05:00Z",
      "acknowledged": false,
      "acknowledged_at": null,
      "acknowledged_by": null,
      "acknowledgement_count": 0
    }
  ]
}

metadata.count is unchanged — this page's item count (data.length). metadata.total is new (Amendment A5) — the count after q/status/ acknowledged filters but before limit/offset, i.e. the total number of matching alerts across every page. Use it to render pagination (page count, "N of M results"), not count.

Once acknowledged, those three fields populate together:

{
  "acknowledged": true,
  "acknowledged_at": "2026-06-30T20:06:00Z",
  "acknowledged_by": { "id": "acct-uuid", "name": "Bernd Hansen" },
  "acknowledgement_count": 1
}

...or, if the acknowledging call was picked up by an external phone contact rather than an account:

{
  "acknowledged": true,
  "acknowledged_at": "2026-06-30T20:06:00Z",
  "acknowledged_by": { "label": "Vendor NOC (Acme)" },
  "acknowledgement_count": 1
}

Show an alert¶

GET /projects/{project_id}/alerts/{id}
Authorization: Bearer <token>

Response 200 — a single alert object (same shape as the list). Response 404 — not found in this project.

Acknowledge an alert (in-app)¶

POST /projects/{project_id}/alerts/{id}/ack
Authorization: Bearer <token>

Halts escalation: acknowledges every one of the alert's currently active escalation instances via the same Alerts::Acknowledge entry point the signed-email link uses. Idempotent — acking an already-acked or already-resolved alert still returns 200.

Response 200 — the updated alert object (acknowledged: true).

Un-acknowledge an alert¶

DELETE /projects/{project_id}/alerts/{id}/ack
Authorization: Bearer <token>

Accidental-click recovery: clears every acknowledgement recorded for this alert, so acknowledged reports false again. Shares the same authorization gate as POST .../ack. Idempotent — un-acking an alert with no acknowledgements still returns 200.

Response 200 — the updated alert object (acknowledged: false).

Acknowledge via signed email link¶

GET  /alerts/ack/{signed_id}
POST /alerts/ack/{signed_id}

The one unauthenticated surface in this API. {signed_id} is a Rails message_verifier-signed, time-limited token minted by the escalation step's email delivery (AlertMailer#ack_link), encoding the alert, the recipient account, and the escalation instance. The signature itself is the authentication — the acknowledgement is attributed to the account embedded in the token, never to a session. Links are valid for 24 hours.

GET returns a confirmation payload for a client to render before committing to the POST:

{
  "metadata": { "organization_id": "org-uuid", "project_id": "proj-uuid", "timestamp": "2026-06-30T20:00:00Z" },
  "data": {
    "alert_id": "uuid",
    "account_id": "uuid",
    "escalation_instance_id": "uuid",
    "alert_summary": "CPU > 90% for 5m",
    "alert_severity": "critical"
  }
}

POST executes the acknowledgement (idempotent):

{
  "metadata": { "organization_id": "org-uuid", "project_id": "proj-uuid", "timestamp": "2026-06-30T20:00:00Z" },
  "data": { "acked": true, "alert_id": "uuid" }
}

Response 401 — invalid_signed_token (tampered) or signed_link_expired (past its 24h validity). Response 404 — the alert or the account embedded in the token no longer exists.

Send a test alert¶

POST /projects/{project_id}/alerts/test
Authorization: Bearer <token>
Content-Type: application/json

Self-serve endpoint (Wave 7) to synthesize and fire a real, firing alert through the platform's actual routing — the exact same seam a real Alertmanager delivery uses (Alert.upsert_from_payload! + Alerts::Router.call, with no test-mode branch or bypass). The fingerprint is always "test-<uuid>" and the alert always carries the label bnerd_test: "true", so it's unambiguous in the UI and easy to exclude from anything that should only count real pages.

{ "route_id": "route-uuid" }

Auto-resolves 10 minutes after firing (TestAlertResolveJob) unless resolved sooner — see Recurring jobs for the job schedule. Resolving early is a plain resolved-status re-post through the same ingest endpoint real Alertmanager deliveries use (POST /projects/{project_id}/alerts with status: "resolved" and the same fingerprint) — there's no dedicated "resolve" action. Either path — the 10-minute auto-resolve or an early manual resolve — produces the full resolved-transition side effects: a resolved timeline event, cancelling any active escalation instance, and the alert_resolved notification endpoint fan-out, identically to a real alert's resolution.

This is a human action, audited normally as alerts.test (unlike the machine POST /alerts ingest endpoint, which explicitly opts out of auditing). Response 201 — the created alert (same shape as list alerts). Response 429 — the per-credential+project throttle above.

Seed recommended defaults¶

POST /projects/{project_id}/alerts/seed_presets
Authorization: Bearer <token>

The "Seed recommended defaults" empty-state action (Wave 7) — creates a starter set of alert routes and escalation chains via Alerts::SeedPresets, for a project that has nothing configured yet.

Response 201

{
  "metadata": { "organization_id": "org-uuid", "project_id": "proj-uuid", "timestamp": "2026-07-03T20:00:00Z" },
  "data": { "routes": 2, "escalation_chains": 1 }
}

routes/escalation_chains are the resulting counts for the project after seeding (not a delta). Response 422 — presets_already_configured.

Alert insights¶

GET /projects/{project_id}/alerts/insights?days=7
Authorization: Bearer <token>

A pure read — alerts:read, not :manage. Six aggregate metrics over a project's alert history in one call: totals, MTTA, daily volume by severity, the noisiest alerts, escalation outcomes, and on-call load — pure SQL aggregation over existing tables, no new tables, strictly scoped to the project.

Response 200

{
  "metadata": { "organization_id": "org-uuid", "project_id": "proj-uuid", "timestamp": "2026-07-03T20:00:00Z" },
  "data": {
    "totals": { "alerts": 42, "firing_now": 3, "acked_share": 0.7143 },
    "mtta": { "median_seconds": 180.0, "p90_seconds": 420.5 },
    "volume": [
      { "date": "2026-07-01", "severities": { "critical": 2, "warning": 5 } },
      { "date": "2026-07-02", "severities": { "critical": 1, "info": 3 } }
    ],
    "top_noisy": [
      { "alertname": "HighCPU", "occurrences": 14 },
      { "alertname": "DiskSpaceLow", "occurrences": 6 }
    ],
    "escalations": {
      "opened": 9, "exhausted": 1, "repeat_rate": 0.2222,
      "acks_by_channel": { "in_app": 5, "email": 2, "phone": 1 }
    },
    "on_call_load": [
      { "account_id": "acct-uuid", "name": "Bernd Hansen", "notified": 12, "called": 2, "acked": 8 }
    ]
  }
}

totals¶

mtta — median/p90 time to first acknowledgement¶

volume — daily counts by severity¶

top_noisy — up to 10 noisiest alert names¶

An entry near the top of this table with a high occurrences count is the platform's signal that a route needs cooldown, a maintenance window, or its matchers tightened — see the guide walkthrough for how to read this table actionably.

escalations¶

on_call_load — per-account notify/call/ack counts¶

Rows come from historical ledger data (timeline events, calls, acknowledgements already recorded at the time they happened) — not a live re-run of on-call resolution — so this reflects who was actually paged/called/acked during the window, including anyone who has since left the roster. See the guide walkthrough for using this as a rotation fairness check.

Response 422 — invalid_days (days present but not 7, 30, or 90).

Alert routes¶

Routes decide, per firing alert, who gets an in-app notification and/or whether an escalation chain opens. A project's routes are evaluated in priority order (ascending); the first route whose matchers all pass wins — there is no accumulation across multiple matching routes.

List alert routes¶

GET /projects/{project_id}/alert_routes
Authorization: Bearer <token>

Returns routes ordered by priority ascending.

Create an alert route¶

POST /projects/{project_id}/alert_routes
Authorization: Bearer <token>
Content-Type: application/json

{
  "name": "Critical DB alerts",
  "priority": 10,
  "matchers": [
    { "label": "severity", "op": "eq", "value": "critical" },
    { "label": "service",  "op": "re", "value": "^db-.*" }
  ],
  "escalation_chain_id": "chain-uuid",
  "in_app_target_type": "org_admins",
  "min_severity_for_notification": "warning",
  "enabled": true
}

Response 201 — the created route. Response 422 — validation error (e.g. an invalid regex in a re matcher, a missing label/value, an in_app_target_type whose companion in_app_target_* field doesn't resolve inside the route's project, the no-op guard below, or duplicate_notification_endpoint_attachment on a concurrent double-attach race).

In-app notification targets¶

in_app_target_type picks who the in-app notification goes to, independent of whether the route also opens an escalation chain:

Record-only: an escalation chain with zero steps¶

An escalation chain with no steps is a deliberate, explicit "record-only" marker, not an oversight — it satisfies the guard above (the route "escalates") while paging nobody: Alerts::Router opens no escalation instance for it (there'd be nothing to advance or time out). Use it for a catch-all route you want to keep in_app_target_type: "none" — e.g. an info-level catch-all that should log the alert but never notify anyone — by pointing it at an otherwise-empty chain instead of leaving the route with no chain at all.

Show / update / delete an alert route¶

GET    /projects/{project_id}/alert_routes/{id}
PATCH  /projects/{project_id}/alert_routes/{id}
DELETE /projects/{project_id}/alert_routes/{id}

PATCH accepts any subset of the create fields. DELETE returns 204 No Content.

Preview a route (dry run)¶

POST /projects/{project_id}/alert_routes/preview
Authorization: Bearer <token>
Content-Type: application/json

{ "labels": { "severity": "critical", "service": "db-primary" } }

A pure read — alert_routes:read, not :write/:manage. Builds an in-memory, never-persisted candidate alert from labels and reports which route would match it, right now, using the exact same AlertRoute.first_match_for the real router uses — no reimplemented matching logic to drift out of sync. Creates no Alert row, enqueues no job, opens no escalation instance, and writes no timeline event.

Response 200

{
  "metadata": { "organization_id": "org-uuid", "project_id": "proj-uuid", "timestamp": "2026-07-03T20:00:00Z" },
  "data": {
    "matched_route": { "id": "route-uuid", "name": "Critical DB alerts", "priority": 10, "...": "..." },
    "would_notify": ["Bernd Hansen", "Verena"],
    "would_escalate": "DB on-call chain",
    "endpoints": [{ "id": "endpoint-uuid", "name": "Slack #db-incidents", "kind": "slack" }],
    "suppressed_by": []
  }
}

Clone a route¶

POST /projects/{project_id}/alert_routes/{id}/clone
Authorization: Bearer <token>

Deep-copies an existing route: name becomes "Copy of {name}", priority is set to the project's current max + 1 (so it sorts to the back of the first-match order and can never silently steal traffic from its source), and it's created disabled. All fields — matchers, severity threshold, in-app target, escalation chain reference, cooldown seconds — and every attached notification endpoint are duplicated. The clone references the same escalation chain / roster / member as its source (not a deep copy of those) — editing the clone's own fields doesn't touch the source, but until you point it elsewhere it still pages/notifies the identical targets.

Response 201 — the cloned route (same shape as create). Response 404 — the source route id isn't in this project.

Notification endpoints¶

A notification endpoint fans an alert out to a machine or a chat tool instead of (or alongside) a human — a monitoring bridge, an internal automation webhook, a Slack channel, a Teams channel. Endpoints are project-scoped resources, created independently of any route, then attached to one or more routes via notification_endpoint_ids. On a route match, the platform delivers to every enabled attached endpoint on both the alert firing and its later resolved transition — an endpoint attached to a route with no in-app target and no escalation chain is a fully valid, standalone routing outcome (it satisfies the "a route must do something" guard on its own).

Kinds¶

slack and teams payloads are pre-formatted for their respective incoming-webhook contract — point url straight at the webhook URL Slack/Teams gives you and there is nothing else to configure on that side. Only webhook endpoints carry a signing secret; slack/teams deliveries are unsigned, matching what those platforms expect.

List / show notification endpoints¶

GET /projects/{project_id}/notification_endpoints
GET /projects/{project_id}/notification_endpoints/{id}
Authorization: Bearer <token>

Neither response includes the secret field — see Create a notification endpoint for the one time it's shown. url is never returned in full: every response carries masked_url instead (everything past the host redacted, e.g. "https://hooks.example.com/…").

Response 200 — show:

{
  "metadata": { "organization_id": "org-uuid", "project_id": "proj-uuid", "timestamp": "2026-07-03T20:00:00Z" },
  "data": {
    "id": "uuid",
    "project_id": "proj-uuid",
    "name": "Ops automation bridge",
    "kind": "webhook",
    "masked_url": "https://hooks.example.com/…",
    "enabled": true,
    "last_delivery": {
      "status": "success",
      "event": "alert_firing",
      "response_code": 200,
      "created_at": "2026-07-03T19:55:00Z"
    },
    "created_at": "2026-07-01T10:00:00Z",
    "updated_at": "2026-07-01T10:00:00Z"
  }
}

last_delivery is null on an endpoint that has never had a delivery attempt.

Create a notification endpoint¶

POST /projects/{project_id}/notification_endpoints
Authorization: Bearer <token>
Content-Type: application/json

{
  "name": "Ops automation bridge",
  "kind": "webhook",
  "url": "https://hooks.example.com/bnerd-alerts",
  "enabled": true
}

Response 201 — a webhook-kind endpoint's response includes secret, the raw HMAC signing key, shown once, here only — it is never returned by any later GET, and a PATCH response shows it again (the same value; PATCH does not rotate it) simply because the endpoint object's full shape is re-rendered. slack/teams endpoints never have a secret (null) — their deliveries aren't signed.

{
  "metadata": { "organization_id": "org-uuid", "project_id": "proj-uuid", "timestamp": "2026-07-03T20:00:00Z" },
  "data": {
    "id": "uuid",
    "project_id": "proj-uuid",
    "name": "Ops automation bridge",
    "kind": "webhook",
    "masked_url": "https://hooks.example.com/…",
    "secret": "3f9c1a...redacted-64-hex-chars",
    "enabled": true,
    "last_delivery": null,
    "created_at": "2026-07-03T20:00:00Z",
    "updated_at": "2026-07-03T20:00:00Z"
  }
}

Response 422 — validation_failed (missing required field, duplicate name in this project, or a url rejected by the SSRF guard).

Update / delete a notification endpoint¶

PATCH  /projects/{project_id}/notification_endpoints/{id}
DELETE /projects/{project_id}/notification_endpoints/{id}
Authorization: Bearer <token>

PATCH accepts any subset of name, kind, url, enabled and, like create, returns the secret again in its response (a webhook endpoint's secret is stable across updates — PATCH never rotates it). DELETE detaches the endpoint from every route it was attached to and returns 204 No Content.

Send a test delivery¶

POST /projects/{project_id}/notification_endpoints/{id}/test
Authorization: Bearer <token>

Synthesizes a sample alert and runs it through the real delivery pipeline — the same formatter, signing, and EndpointDelivery bookkeeping a live alert would use, with event: "test" — so a successful test-send is a genuine end-to-end proof the endpoint is reachable and correctly configured, not a synthetic ping that bypasses the real code path.

Response 202 — the test delivery is enqueued asynchronously, not delivered synchronously in this response; check deliveries or the endpoint's last_delivery a moment later for the outcome.

{
  "metadata": { "organization_id": "org-uuid", "project_id": "proj-uuid", "timestamp": "2026-07-03T20:00:00Z" },
  "data": { "ok": true, "test_alert_id": "uuid" }
}

Recent deliveries¶

GET /projects/{project_id}/notification_endpoints/{id}/deliveries
Authorization: Bearer <token>

Newest first, capped at the 50 most recent delivery attempts.

Response 200

{
  "metadata": { "organization_id": "org-uuid", "project_id": "proj-uuid", "timestamp": "2026-07-03T20:00:00Z", "count": 1 },
  "data": [
    {
      "id": "uuid",
      "alert_id": "uuid",
      "event": "alert_firing",
      "status": "failed",
      "response_code": null,
      "attempts": 5,
      "last_error": "Faraday::ConnectionFailed: execution expired",
      "created_at": "2026-07-03T19:55:00Z"
    }
  ]
}

The webhook JSON payload¶

A webhook-kind endpoint's body (slack/teams bodies are their own platform's native shape, not this one):

{
  "event": "alert_firing",
  "alert": {
    "id": "uuid",
    "fingerprint": "a1b2c3d4e5f6",
    "status": "firing",
    "severity": "critical",
    "labels": { "alertname": "HighCPU", "severity": "critical", "instance": "web-01" },
    "annotations": { "summary": "CPU > 90% for 5m" },
    "starts_at": "2026-07-03T19:55:00Z",
    "ends_at": null,
    "last_received_at": "2026-07-03T19:55:00Z"
  },
  "project": { "id": "proj-uuid", "name": "production" },
  "alert_url": "https://app.bnerd.cloud/projects/proj-uuid/alerts/uuid",
  "sent_at": "2026-07-03T19:55:01Z"
}

event is alert_firing or alert_resolved for a real delivery, test for a test delivery — the shape is otherwise identical.

Verifying webhook signatures¶

Every webhook-kind delivery carries three headers a receiver can use to authenticate and de-duplicate it:

To verify a delivery: compute HMAC-SHA256(secret, raw_request_body) yourself over the exact bytes of the body as received (don't re-serialize parsed JSON — whitespace/key-order differences will break the signature), hex-encode it, prefix with sha256=, and compare to the X-Bnerd-Signature header using a constant-time comparison — never == on the two strings, which leaks timing information an attacker can use to forge a valid signature byte by byte.

# Ruby / Rails example
expected = "sha256=" + OpenSSL::HMAC.hexdigest("SHA256", endpoint_secret, raw_body)
ActiveSupport::SecurityUtils.secure_compare(expected, request.headers["X-Bnerd-Signature"])

# Python example
import hmac, hashlib

expected = "sha256=" + hmac.new(endpoint_secret.encode(), raw_body, hashlib.sha256).hexdigest()
hmac.compare_digest(expected, request.headers["X-Bnerd-Signature"])

slack/teams deliveries carry no signature header — Slack's and Teams' own incoming-webhook URLs are the shared secret for those kinds (the URL itself is unguessable and should be treated as sensitive), so there's nothing further to verify on receipt.

Retries¶

A failed delivery attempt is classified transient or permanent:

Once a delivery is exhausted (5 transient attempts, or hits a permanent classification on any attempt), it stays failed — there is no automatic re-drive. Send a test delivery after fixing the underlying issue to confirm the endpoint works again.

SSRF protection¶

Endpoint URLs are validated twice: once at save time (create/update), and again immediately before every single delivery attempt — because DNS can be re-pointed to a private address in the window between the two, a save-time-only check would not be sufficient. A URL is rejected (422 at save; recorded as a permanent delivery failure at send time) if it:

• isn't a valid http(s) URL, or uses plain http outside a dev/test environment;
• fails to resolve at all; or
• resolves to a private, loopback, link-local, carrier-grade-NAT, unique-local, multicast, or unspecified address — including the 169.254.169.254 cloud-metadata address specifically.

The outbound connection is then pinned to the exact address that validation just checked, rather than re-resolving the hostname a second time independently — closing the DNS-rebinding gap where a second lookup at connect-time could return a different (private) address than the one that was validated. Redirects are never followed for the same reason: a 3xx to a private target would otherwise bypass the guard entirely.

Attaching endpoints to escalation steps¶

A route's notification_endpoint_ids (above) fans out flat and immediate — every match, right away. An escalation step can attach its own endpoint set the same way, but staged in time with the rest of the chain — the natural tool for widening the blast radius on continued non-acknowledgement: page the on-call roster on step 1, and only post to an incident-response chat channel or hand off to an automation webhook on step 2 if nobody's acked by then. Route and step attachment are complementary, not redundant — use a route endpoint for "always," a step endpoint for "only if this drags on."

POST /projects/{project_id}/escalation_chains/{chain_id}/escalation_steps
{
  "step": {
    "position": 2,
    "target_type": "org_admins",
    "channels": ["in_app", "email"],
    "delay_seconds": 0,
    "ack_timeout_seconds": 600,
    "notification_endpoint_ids": ["<incident-response-webhook-id>"]
  }
}

Same field, same replace-the-full-set semantics as a route: send every id you want attached ([] detaches all, omitting the key on PATCH leaves attachments unchanged), and ids that don't resolve to a NotificationEndpoint in the chain's project are silently dropped. A step's show/index response includes its notification_endpoint_ids.

Attached endpoints fire on every genuine step run, alongside the step's other channels, and are silence-suppressed exactly like a step's human channels: a silenced alert's step still advances the chain on schedule, but pages neither its people nor its attached endpoints.

A step's endpoint fan-out is recorded as a notified timeline event with channel: "endpoint" — not the route-level endpoint_dispatched event kind, since a step's endpoints fire time-staged with the rest of the step rather than immediately on route match.

Escalation chains¶

A chain is an ordered list of steps plus the re-escalation and repeat behavior around them.

List / show escalation chains¶

GET /projects/{project_id}/escalation_chains
GET /projects/{project_id}/escalation_chains/{id}

show includes the chain's steps, ordered by position.

Create an escalation chain¶

POST /projects/{project_id}/escalation_chains
Authorization: Bearer <token>
Content-Type: application/json

{
  "name": "Production on-call",
  "enabled": true,
  "re_escalation_after": 1800,
  "repeat_mode": "n_times",
  "max_repeats": 3,
  "repeat_delay_seconds": 300
}

Response 201 — the created chain (with an empty steps: []). Response 422 — e.g. repeat_mode: "n_times" without max_repeats.

Update / delete an escalation chain¶

PATCH  /projects/{project_id}/escalation_chains/{id}
DELETE /projects/{project_id}/escalation_chains/{id}

PATCH accepts any subset of the create fields. DELETE also deletes the chain's steps and returns 204 No Content.

Clone a chain¶

POST /projects/{project_id}/escalation_chains/{id}/clone
Authorization: Bearer <token>

Deep-copies an existing chain: name becomes "Copy of {name}", created disabled, and every step is duplicated in order — same target_type/target reference, channels, delay_seconds, ack_timeout_seconds, repeat, and any attached notification endpoints. A cloned step still pages the same roster/member/org-admins as its source until you edit it — cloning duplicates the chain's shape, not a fresh, blank set of targets.

Response 201 — the cloned chain, with its cloned steps (same shape as create). Response 404 — the source chain id isn't in this project.

Who's on call right now (resolved targets)¶

GET /projects/{project_id}/escalation_chains/{id}/resolved_targets
Authorization: Bearer <token>

A pure read — escalation_chains:read, not :manage — reports, per step, exactly who EscalationStep#resolve_targets would page right now if this chain were actively escalating. This is not a reimplementation: it calls the identical resolution method the paging engine itself uses, so it can never drift out of sync with what actually happens during a real page. Rotation-aware, and absence-aware — an absent account is excluded exactly as it would be during real paging, including falling back to every roster member if absence would otherwise empty the resolved set (see Absence exclusion).

Response 200

{
  "metadata": { "organization_id": "org-uuid", "project_id": "proj-uuid", "timestamp": "2026-07-03T20:00:00Z" },
  "data": {
    "escalation_chain_id": "chain-uuid",
    "steps": [
      {
        "step_id": "step-uuid",
        "position": 1,
        "target_type": "roster",
        "accounts": [{ "id": "acct-uuid", "name": "Bernd Hansen" }]
      },
      {
        "step_id": "step-uuid-2",
        "position": 2,
        "target_type": "org_admins",
        "accounts": []
      }
    ]
  }
}

accounts is id/name only — never email or a phone number, narrower than the roster listing endpoint's own PII surface. An empty accounts array on a step is a legitimate result ("nobody currently resolves" for that step, e.g. an empty roster with no rotation and no members) — not an error, and distinct from the fallback case where an otherwise-empty resolution instead returns every roster member. Response 404 — the chain id isn't in this project.

Escalation steps¶

Steps are nested under a chain and always operate on a step root key in the request body — not flat top-level fields. (action is a Rails-reserved top-level param name that would otherwise be silently overwritten by the routing action; nesting under step avoids the collision.)

List / show escalation steps¶

GET /projects/{project_id}/escalation_chains/{escalation_chain_id}/escalation_steps
GET /projects/{project_id}/escalation_chains/{escalation_chain_id}/escalation_steps/{id}

Returned ordered by position ascending.

Add a step to a chain¶

POST /projects/{project_id}/escalation_chains/{escalation_chain_id}/escalation_steps
Authorization: Bearer <token>
Content-Type: application/json

{
  "step": {
    "position": 1,
    "target_type": "roster",
    "target_roster_id": "roster-uuid",
    "channels": ["in_app", "email"],
    "delay_seconds": 0,
    "ack_timeout_seconds": 300,
    "repeat": 1
  }
}

Response 201 — the created step. Response 422 — e.g. target_type: "member" with no target_member_id, or a target that doesn't belong to the chain's project.

Update / delete an escalation step¶

PATCH  /projects/{project_id}/escalation_chains/{escalation_chain_id}/escalation_steps/{id}
DELETE /projects/{project_id}/escalation_chains/{escalation_chain_id}/escalation_steps/{id}

PATCH body is the same step:-nested shape (any subset of fields). DELETE returns 204 No Content.

On-call rosters¶

A roster is a named group of accounts a step can target as a unit.

List / show on-call rosters¶

GET /projects/{project_id}/on_call_rosters
GET /projects/{project_id}/on_call_rosters/{id}

show includes the roster's members (each { id, email, name }, plus an absent flag — see Absence exclusion below).

Create / rename / delete an on-call roster¶

POST   /projects/{project_id}/on_call_rosters
PATCH  /projects/{project_id}/on_call_rosters/{id}
DELETE /projects/{project_id}/on_call_rosters/{id}

{ "name": "Platform on-call" }

PATCH accepts { "name": "..." }. DELETE returns 204 No Content.

Add / remove roster members¶

GET    /projects/{project_id}/on_call_rosters/{on_call_roster_id}/on_call_roster_members
POST   /projects/{project_id}/on_call_rosters/{on_call_roster_id}/on_call_roster_members
DELETE /projects/{project_id}/on_call_rosters/{on_call_roster_id}/on_call_roster_members/{id}

{ "account_id": "acct-uuid" }

The account must already hold a project membership in this project — pages must never reach someone with no relationship to it. A duplicate add (already on the roster) and a non-project-member account both fail with 422, never 500.

Response 201 — the roster member ({ id, on_call_roster_id, account_id, account, created_at }). Response 204 — member removed.

External phone contacts¶

A roster member (above) must already hold a project membership — but a phone-channel step often also needs to call people with no b'nerd account at all: an external NOC hotline, a vendor's own on-call line. A phone contact covers that case — a labeled phone number attached to a roster that only ever participates in phone-channel paging.

GET    /projects/{project_id}/on_call_rosters/{on_call_roster_id}/on_call_roster_phone_contacts
POST   /projects/{project_id}/on_call_rosters/{on_call_roster_id}/on_call_roster_phone_contacts
DELETE /projects/{project_id}/on_call_rosters/{on_call_roster_id}/on_call_roster_phone_contacts/{id}

{ "label": "Vendor NOC (Acme)", "phone": "+493012345678", "consent": true }

consent must be truthy — the server stamps consent_acknowledged_at from the current time (any client-supplied timestamp is ignored) and records created_by_id from the acting account. Missing/false consent is 422 consent_required; a duplicate phone on the same roster, or an invalid E.164 number, is 422 — never a 500. DELETE removes the contact from every future call and returns 204 No Content.

Response 200/201 — { id, on_call_roster_id, label, phone, phone_masked, consent_acknowledged_at, created_by_id, created_at }. Both phone (E.164) and phone_masked (the roster UI's display form, e.g. "+49••••5678") are visible to any project member holding alerts:read; the write payload only ever accepts phone. The static-token scope is on_call_roster_phone_contacts (read/write/manage).

How a contact participates in paging:

• Only the phone channel resolves a roster's contacts — a roster target's in_app/email delivery stays account-only, since a contact has no account to notify in-app or an inbox to email.
• Unlike an account, a contact is never gated on phone verification — see DSGVO: consent replaces verification below.
• A contact acknowledges exactly like an account: pressing 1 on the call. Because it has no account, its acknowledgement is attributed by label instead of { id, name } — see acknowledged_by above and the acked/call_placed rows in Event kinds.

DSGVO: consent replaces verification¶

• Consent is explicit and recorded, not inferred. Creating a contact requires an affirmative consent: true; the platform stamps who added the contact (created_by_id) and when consent was acknowledged (consent_acknowledged_at) — both are part of the resource and auditable like any other record.
• Purpose-limited storage. The number is stored solely to place escalation calls to it. It's visible only to project members holding alerts:read, and deletable at any time — deleting a contact removes it from every future call immediately; there is no soft-delete or retention window.
• No verification call. Recorded consent is the sole lawful basis for calling the number — a contact never receives the phone-verification call an account does.
• Minimal call metadata, no recordings. Only call status (queued, answered, no-answer, failed, ...) is logged for a contact's calls, the same as for an account's; the platform never records call audio.
• Redacted everywhere else. A contact's raw number never lands in the clear in an alert timeline event (masked to the first six characters, same as an account's — see call_placed in Event kinds) or in the platform's audit log — phone is a filtered parameter, so a create/update request body is redacted before it's ever logged.

On-call schedules & rotations¶

A roster's static membership list (above) is always the ultimate fallback, but a roster can optionally layer time-based scheduling on top of it: one rotation that hands off between its participants on a repeating cadence, plus any number of ad-hoc overrides for one-off coverage swaps. Both are fully optional and fully backward compatible — a roster with neither keeps paging its whole static membership list exactly as before this feature shipped.

Rotation¶

A roster has at most one rotation, enforced by a DB unique index — a second POST for an already-covered roster fails 422, never 500. It's a nested singular resource (.../rotation, no {id}), matching the "one rotation per roster" cardinality.

GET    /projects/{project_id}/on_call_rosters/{roster_id}/rotation
POST   /projects/{project_id}/on_call_rosters/{roster_id}/rotation
PATCH  /projects/{project_id}/on_call_rosters/{roster_id}/rotation
DELETE /projects/{project_id}/on_call_rosters/{roster_id}/rotation
Authorization: Bearer <token>

{
  "rotation_type": "weekly",
  "handoff_time": "09:00",
  "handoff_weekday": 1,
  "timezone": "Europe/Berlin",
  "anchor_date": "2026-07-06",
  "enabled": true,
  "participants": [
    { "account_id": "acct-a-uuid", "position": 0 },
    { "account_id": "acct-b-uuid", "position": 1 }
  ]
}

Response 201/200 — the rotation, with its resolved participants (each carrying the nested account summary plus an absent flag — see Absence exclusion below). Response 204 — deleted. Response 422 — validation_failed (bad timezone, malformed handoff_time, missing interval_days on a custom rotation, an already-existing rotation on POST) or duplicate_participant.

Overrides¶

An override is an ad-hoc, time-windowed coverage swap — sick day, planned trade, whatever doesn't belong in the standing rotation. There's no :show/:update: an override is create-or-destroy only — replace one by deleting it and creating a new one.

GET    /projects/{project_id}/on_call_rosters/{roster_id}/on_call_overrides
POST   /projects/{project_id}/on_call_rosters/{roster_id}/on_call_overrides
DELETE /projects/{project_id}/on_call_rosters/{roster_id}/on_call_overrides/{id}
Authorization: Bearer <token>

{
  "account_id": "acct-covering-uuid",
  "starts_at": "2026-07-10T00:00:00Z",
  "ends_at": "2026-07-11T00:00:00Z",
  "reason": "Swap with Bernd"
}

created_by is stamped server-side from the authenticated caller and is never client-settable. Response 201 — the override, including created_by_id. Response 204 — override removed.

Two or more overrides can be simultaneously active on the same roster — they union, rather than one silently winning: every account covered by an active override at a given instant is paged.

Resolution rules¶

Who a roster actually pages at a given instant at (used for both live paging and the roster's current_on_call field) is decided in this order:

1. Active overrides win. If one or more overrides cover at, exactly those accounts page — the computed rotation (if any) is bypassed entirely for the window. Simultaneous overrides union (see above).
2. Else the enabled rotation's scheduled participant. If the roster has an enabled: true rotation and it resolves to a participant at at, that one account pages.
3. Else all roster members. A roster with no rotation, or a disabled one, pages its full static membership list — unchanged behavior for every roster that predates this feature.
4. Never page nobody. If rule 2 was expected to apply (an enabled rotation exists) but it resolves to no one — no participants configured yet, or at precedes the rotation's very first handoff — the platform falls back to paging every roster member (same set as rule 3) instead of paging no one, and records an on_call_fallback timeline event on the alert being paged. The same fallback (without a timeline event, since there's no alert in hand) also covers a roster with no rotation and no members at all.

A roster's GET/show/index payload includes a current_on_call field — the accounts these rules resolve to right now — alongside the always-full members list.

Absence exclusion¶

Amendment A1: before rules 1–3 above are applied, accounts with an active absence at at are excluded from whatever set the rule would otherwise resolve — this runs before the never-page-nobody fallback (rule 4), not after. Absence does not cascade across rules: an absent override person does not fall through to the rotation or the plain-members rule, it's simply removed from the override's own resolved set.

• If excluding absent accounts still leaves at least one account, that (smaller) set pages — no fallback triggered.
• If excluding absent accounts would empty the resolved set entirely, the roster falls back to every roster member, exactly like rule 4 — an all-absent roster still pages, it never pages nobody.
• If the roster has no members at all to fall back to (an active override or rotation participant for someone who isn't on the roster is a real, reachable state), the fallback pages the absent person(s) instead of returning nobody — paging someone marked away beats paging no one.

This is reflected in the roster and rotation payloads so a dashboard can render an "away" indicator without re-deriving resolution logic itself:

GET .../shifts (below) has no absent field — a future/past shift segment has no single "now" to evaluate absence against.

The shifts endpoint¶

GET /projects/{project_id}/on_call_rosters/{id}/shifts?from&to
Authorization: Bearer <token>

A server-computed, calendar-ready schedule for a roster — rotation segments and override segments overlapping [from, to], spliced together exactly the way resolution would decide them at any instant in the window, so a dashboard's upcoming-shifts strip can never disagree with who actually gets paged. from/to are optional date-time query params (default now / now + 7 days); a present-but-unparseable value is a 422 invalid_window, never a silent fallback to the default.

Rotation segments are clipped against any overlapping override windows (an active override always outranks the rotation — rule 1 above), so the returned segments never disagree about who covers a given instant. Overrides themselves may still overlap each other in the response (consistent with their union semantics). The result is sorted by starts_at and capped at 100 segments.

Response 200

{
  "metadata": { "organization_id": "org-uuid", "project_id": "proj-uuid", "timestamp": "2026-07-03T20:00:00Z", "count": 2 },
  "data": [
    {
      "source": "rotation",
      "starts_at": "2026-07-06T07:00:00Z",
      "ends_at": "2026-07-10T00:00:00Z",
      "account": { "id": "acct-a-uuid", "name": "Alex", "email": "alex@example.com" }
    },
    {
      "source": "override",
      "starts_at": "2026-07-10T00:00:00Z",
      "ends_at": "2026-07-11T00:00:00Z",
      "account": { "id": "acct-covering-uuid", "name": "Bernd Hansen", "email": "bernd@example.com" }
    }
  ]
}

Response 422 — invalid_window (from or to present but not a parseable timestamp).

Silences¶

A silence suppresses paging for one alert while it is active — the escalation state machine keeps advancing quietly underneath (steps still time out and advance/exhaust/loop), it just delivers no notifications while silenced. V1 is per-alert only; matcher-based silences are deferred to a future release.

List silences for an alert¶

GET /projects/{project_id}/alerts/{alert_id}/silences
Authorization: Bearer <token>

Most recent first.

Silence (snooze) an alert¶

POST /projects/{project_id}/alerts/{alert_id}/silences
Authorization: Bearer <token>
Content-Type: application/json

Accepts either an explicit expires_at or a duration_seconds convenience (computed relative to request time) — at least one is required.

{ "duration_seconds": 1800, "reason": "Known noisy alert during deploy" }

Resumes automatically on expiry if the alert is still firing — no manual action needed to un-silence.

Response 201

{
  "metadata": { "organization_id": "org-uuid", "project_id": "proj-uuid", "timestamp": "2026-06-30T20:00:00Z" },
  "data": {
    "id": "uuid",
    "alert_id": "uuid",
    "expires_at": "2026-06-30T20:30:00Z",
    "reason": "Known noisy alert during deploy",
    "created_by": { "id": "acct-uuid", "name": "Bernd Hansen" },
    "created_at": "2026-06-30T20:00:00Z",
    "updated_at": "2026-06-30T20:00:00Z"
  }
}

Lift a silence early¶

DELETE /projects/{project_id}/alerts/{alert_id}/silences/{id}
Authorization: Bearer <token>

Response 204 — silence lifted.

Maintenance windows¶

A maintenance window is a scheduled, matcher-based suppression — the scheduled counterpart to a silence's ad-hoc, per-alert one. While a window is active and its matchers match an alert's labels, Alerts::Router suppresses all paging for that alert: no in-app/email notify, no escalation instance opened (or re-opened on recurrence), and no notification endpoint fan-out on either alert_firing or alert_resolved. It's checked first, ahead of the silence check, so an active window wins regardless of any per-alert silence state. See Noise control: maintenance windows for when to reach for a window versus a silence versus cooldown.

GET    /projects/{project_id}/maintenance_windows
GET    /projects/{project_id}/maintenance_windows/{id}
POST   /projects/{project_id}/maintenance_windows
PATCH  /projects/{project_id}/maintenance_windows/{id}
DELETE /projects/{project_id}/maintenance_windows/{id}
Authorization: Bearer <token>

{
  "name": "DB migration — 2026-07-10",
  "matchers": [
    { "label": "service", "op": "re", "value": "^db-.*" }
  ],
  "starts_at": "2026-07-10T22:00:00Z",
  "ends_at": "2026-07-10T23:30:00Z",
  "note": "Planned schema migration, expect noisy replication-lag alerts"
}

Response 200/201 — the window:

{
  "metadata": { "organization_id": "org-uuid", "project_id": "proj-uuid", "timestamp": "2026-07-03T20:00:00Z" },
  "data": {
    "id": "uuid",
    "name": "DB migration — 2026-07-10",
    "matchers": [{ "label": "service", "op": "re", "value": "^db-.*" }],
    "starts_at": "2026-07-10T22:00:00Z",
    "ends_at": "2026-07-10T23:30:00Z",
    "note": "Planned schema migration, expect noisy replication-lag alerts",
    "created_by": { "id": "acct-uuid", "name": "Bernd Hansen" },
    "created_at": "2026-07-03T20:00:00Z",
    "updated_at": "2026-07-03T20:00:00Z"
  }
}

Response 204 — window deleted. Response 422 — validation_failed (ends_at not after starts_at, or missing name).

Alert timeline¶

Every lifecycle transition on an alert — fired, routed, escalated, notified, acknowledged, silenced, and more — is recorded to an append-only ledger. It's what powers the alert detail page's history view.

List an alert's timeline¶

GET /projects/{project_id}/alerts/{alert_id}/alert_events
Authorization: Bearer <token>

Returns events newest-first.

RBAC runs through the same alerts permission category as the alert itself (read level), evaluated against the specific alert instance — not a project-wide check. The static-token scope is alert_events (see Authorization); read is the only meaningful level, since there is no mutating action on this resource.

Response 200

{
  "metadata": { "organization_id": "org-uuid", "project_id": "proj-uuid", "timestamp": "2026-07-03T20:00:00Z", "count": 2 },
  "data": [
    {
      "id": "uuid",
      "kind": "acked",
      "occurred_at": "2026-07-03T20:06:00Z",
      "account": { "id": "acct-uuid", "name": "Bernd Hansen" },
      "escalation_instance_id": "ei-uuid",
      "payload": { "account_name": "Bernd Hansen", "via_channel": "in_app" }
    },
    {
      "id": "uuid",
      "kind": "step_started",
      "occurred_at": "2026-07-03T20:05:00Z",
      "account": null,
      "escalation_instance_id": "ei-uuid",
      "payload": { "position": 1, "channels": ["in_app", "email"], "target_names": ["Platform on-call"] }
    }
  ]
}

Event kinds¶

Genuine-transitions guarantee¶

Every row is written through a single internal choke point (Alerts::Timeline.record!) — never directly by a controller or job — which only ever appends on a genuine state transition, never on a replay:

• A still-firing Alertmanager redelivery of an alert with an already-active escalation instance does not re-emit fired/escalation_opened.
• A double ack click (or two callers racing to ack the same alert) emits acked once, only on the call that actually halted an active instance or created the alert's first-ever acknowledgement record — not once per halted instance, and not on a pure replay.
• A redelivered Twilio status callback carrying a status the platform already recorded emits nothing; call_status only writes when the incoming status differs from the stored one.

Like the platform's notification delivery, a timeline write is best-effort: a failure to write an event is logged and swallowed, never allowed to break the ingest/routing/escalation call that's emitting it.

Retention¶

Timeline events are pruned after 90 days by a daily job. Once pruned, they're gone — there is no archive or export endpoint in V1.

Error responses¶

Errors follow the standard error envelope. Alerting-specific codes: