Skip to main content

​
On-call roster & escalation policy runbook

The On-Call API flattens a roster and a policy into one absolute timeline of pages: which targets, on which channels, at which offsets from the incident start. This runbook is the operator’s walk through that timeline — how to model the roster, how step timeouts accrue, what happens when the primary misses the page, and what you owe the timeline when it runs out. For the endpoint-by-endpoint build, see the on-call alerting guide; for the design rationale, see the escalation model concept page.
Every On-Call endpoint is pure compute over the request body — no state is stored on Orbit’s side and nothing is sent. You keep the incident snapshot in your own store; the page itself exits through the Messages API, where your sender ids and quiet-hours policy apply. All knobs below — roster membership, timeouts, channels, thresholds — are controls your tenant owns.

​
Section 1 — Rosters and rotation groups

A roster is a rotation: an ordered members list, a cadence (daily, weekly, or custom with shift_length_seconds), and an anchor — the absolute ISO instant member[0]‘s first shift begins. Anchoring to an absolute instant is what keeps the rotation DST-proof: pick the instant once, in whichever timezone you like, and every hand-off after that is exactly one shift length later. There is no calendar attachment to configure; the anchor is the calendar, and real roster entries are the ordered member ids you supply. Members are opaque strings — user ids, E.164 numbers, emails, push tokens; the engine picks which one, never interprets the value. Before the anchor, resolve still names a human: member[0] holds the notional shift that ends at the anchor, so a query that lands before the schedule starts is never pager-less. A backdated at returns the historically correct holder, which is how you answer “who was on call when this fired.”

​
Section 2 — Escalation policies: timeouts, repeat, override

A policy is an ordered steps list. Each step names a target (a live rotation, resolved at page time, or a fixed users list), the channels to page, and escalate_after_seconds — how long to wait for an acknowledgement before the next step loads. Offsets accrue as a cumulative sum: step 0 fires at offset 0, step 1 at offset step0.escalate_after_seconds, and so on. repeat loops the whole policy that many additional times when nobody acknowledges (PagerDuty’s “repeat policy N times” knob). POST /api/v1/oncall/escalation/plan returns the full timeline up front — every page’s round, step, offset_seconds, fire_at, targets, and channels, plus total_pages across all rounds. Preview the plan before an incident depends on it. The one override path is reassign: an operator action that substitutes a new policy, resets the high-water mark to zero, and restarts the timeline from step 0 — the clean way to bump a Sev-2 onto a broader Sev-1 policy mid-incident. ack halts escalation; resolve closes it. Invalid transitions (ack a resolved incident, reassign after resolution) are rejected with 409 incident_transition_invalid instead of silently corrupting your snapshot.

​
Section 3 — Alert channels

Each step lists one or more channels from the Orbit channel set — for paging duty that typically means push and sms on the early, cheap steps, with voice reserved for the fallback (a voice call cuts through a muted phone in a way a text does not). The channel list is declarative; the actual send is yours to make. When tick returns due pages, fan each one out through the Messages API on the channels the step named, so sender ids, quiet-hours policy, and delivery webhooks apply to a page exactly as they do to any other send. Subscribe to delivery status so “the page never landed” is distinguishable from “they never acked” before the step window elapses.

​
Section 4 — Driving the incident: acknowledge, resolve, assign

You own the incident record: a snapshot with its policy, started_at, current status (open → acknowledged → resolved), and a pages_fired high-water mark. Because the timeline fires in strictly increasing order, one integer is a complete progress record — replaying a tick with a stale snapshot re-returns the same due pages, never a double page.
  • POST /api/v1/oncall/incident/tick — what’s due now: the pages to send, the new pages_fired mark to persist, and next_tick_at, when to call again. A halted (acknowledged/resolved) incident returns due_pages: [] and next_tick_at: null, so your scheduler stands down.
  • POST /api/v1/oncall/incident/transition — apply ack { by }, resolve, or reassign { policy }. Persist the returned snapshot after every call.
The dashboard carries the same preview: Voice → On-call & escalation runs resolve and escalation/plan against the rotation and policy you paste in, so you can check who holds the pager and what the timeline looks like before you wire an incident driver to it.

​
Section 5 — Tuning thresholds, silence, and cooldown

Three knobs tune the alarm:
  • escalate_after_seconds per step (≤ 24h; default 5 min) — the acknowledge window. Too short and you train responders to ignore pages that have already escalated; too long and a missed acknowledgement burns MTTR. Five to fifteen minutes per step is the usual band.
  • repeat (0–10) — policy-level cooldown. What to do when the whole chain fires and nobody answers: re-run the chain instead of going silent.
  • Exhaustion — when every page across every round has fired with no ack, tick reports status: "exhausted" and next_tick_at: null. Treat exhaustion as its own alert — page a fallback channel or open a ticket — rather than keep a scheduler awake on a null next-tick. That terminal signal is what an intentionally silenced alarm still owes you: either it acknowledges, or it surfaces as exhausted.

​
Worked example — Sev-1 chain with a missed primary

Two steps, repeat: 1 — four pages in the timeline, six minutes from first page to exhaustion: At offset 0 the primary is paged over push and SMS. If no ack lands within the 300-second window, the timeline hands the page to usr_9 over voice — the missed-primary override. Rotation targets are re-resolved at each page’s fire instant, so a hand-off that lands mid-incident pages whoever holds the pager then. An ack arriving at any point halts the rest; if nothing ever acknowledges, the second round re-runs the chain and the incident reports exhausted — which you treat as its own alarm rather than silence.

​
See also