Skip to main content

​
Incident-day operations: failover drills, status surfaces, and tenant notifications

Orbit ships the controls an operator needs across a carrier or region incident: failover you can rehearse, diagnostics that name the failing step, org-scoped status views, notification fan-out, and tenant-owned levers to pause, reroute, and recover traffic. Those controls are documented on their own pages; this guide walks them in the order an operator uses them: rehearse, detect, steer, recover and report. Everything on this page is tenant-owned. You operate your trunks, routes, pools, and pauses; Orbit operates the platform. Nothing here promises platform availability; for the platform’s own declarations, read Settings Status and the platform status page.

​
Part 1: Rehearse failover before you need it

A failover path you have never watched walk is a failover path you do not trust at 2 a.m. Rehearse both sides, voice and messaging, against a non-production organization or a staging SBC.

​
The SIP trunk failover drill

The mechanics are documented in SIP trunk connection: health checks and failover; this is the drill built on them.
  1. Create two outgoing trunks against your staging equipment: trunk A (primary) and trunk B (fallback).
  2. Set failoverTrunkId on trunk A, pointing at trunk B. An update that would create a loop is rejected with a 422, so chains stay walkable. An incoming trunk is a valid fallback target too: its first allowed IP is your own already-validated SBC address.
  3. Baseline both trunks. Run Test trunk (POST /api/v1/voice/sip-trunks/{id}/test) on each and expect status: "registered" with a non-null latency. The same probe runs continuously in the background, so the trunk row also carries lastRegisteredAt and lastRegistrationError.
  4. Kill trunk A on purpose: stop the staging registrar, or point trunk A’s host at an endpoint that will never answer a REGISTER. Run Troubleshoot on A and expect the register step to read failed (definitive rejection) or pending (no answer within the probe budget), with every earlier step ok. The fleet health snapshot (GET /api/v1/voice/sip-trunks/health) now ranks A unregistered, needs-attention first.
  5. Place a test call through your normal outbound path. A dispatch that finds the primary trunk unregistered walks the failover chain first, then falls back to the platform default, keeping the call down instead of dropping it.
  6. Verify the CDR shows the fallback leg on trunk B, then restore trunk A and confirm it re-registers before you call the drill done.
Outbound PSTN egress always stays on Orbit’s wholesale path; your trunks connect your equipment to Orbit and never carry the platform’s outbound exit. The drill above rehearses delivery into and out of your own trunks, exactly the surface a carrier or region incident touches.

​
The messaging cascade drill

The messaging-side equivalent is a cascade group: a primary send plus every fallback hop, bound to one message_group_id (Cascade failover policy and logical cascade groups).
  1. On a sandbox key (dv_test_sk_), send with a typed cascade chain, for example a primary channel with "cascade": { "channels": ["sms"] }.
  2. Force the primary leg to fail using the sandbox magic numbers (Sandbox magic numbers playbook): a recipient whose trailing digit is 3 terminates undelivered and a trailing 4 terminates failed, replaying the same webhook sequence a real carrier produces.
  3. Read the group back with GET /api/v1/messages/cascade/groups/<group_id>. The drill passes when the response shows the primary leg terminal and a second leg carrying fallback_of (the primary’s id) and fallback_reason (for example primary_rcs_undelivered), with leg_count matching the hops that fired. That is the chain walking, end to end, with no carrier involved.
The sandbox short-circuits before any provider is touched and bills nothing, so this drill is safe to run as often as you like. The same rehearsal works against an org-level cascade policy (PUT /api/v1/messages/cascade/policy) when you want every send to escalate through the same chain.

​
Part 2: Detect: which surface shows what

During a real incident, four surfaces answer four different questions. Read them in this order.

​
Trunk diagnostics

The diagnostics panel’s ordered steps (resolve, credentials, provision, gateway, register) name the phase that broke, so a carrier incident reads differently from a misconfiguration: a registrar that was healthy an hour ago and now returns register failed or pending across every trunk pointed at it is a carrier story, while a resolve failure is usually your host field. Incoming trunks have no REGISTER to place; they track accepted-traffic freshness instead (registered within 5 minutes, trying within an hour, else idle) and surface rejections on lastCallVerdict with the exact reason. The field-by-field reading lives in SIP trunk health and connectivity diagnostics.

​
Route quality and the circuit breaker

For SMS and MMS, each destination operator carries a 0–100 health score folded into a band: healthy at 85 and above, degraded from 60 to 84, poor from 35 to 59, critical below. A route sitting in the degraded band with a climbing latency_p95_ms is the early-warning shape of a carrier incident; a collapsed delivery_rate with a high ack_no_receipt_rate is the grey-route shape instead. Your thresholds decide what counts as a breach, and if you opted in, the circuit breaker auto-suspends a sustained-failing route and fails traffic over until it recovers. The full workflow is the route-quality circuit breaker runbook.

​
The org status console

Settings → Status maps declared platform incidents to the channels your organization actually uses. Its incident states follow the published progression (Investigating, Identified, Monitoring, Resolved), and severity rolls up worst-first across overlapping records. A channel touched by an active incident is presented at that incident’s severity; that is what a marked channel looks like on this surface. Conversely, an incident that does not map to any of your channels never appears in your org history, and the absence of an incident is not proof your channel is healthy: tenant-level delivery failures (carrier rejection, filtered content, insufficient balance) degrade while platform availability reads 100%. The reading order is in Settings Status; the projection model behind it is the org-scoped status model.

​
Notifications

During an incident the kinds that fire are mostly alerts (a wallet that hit zero and paused outbound, a spend threshold that tripped) and anomalies (a webhook endpoint auto-disabled after repeated failures, a DNS regression that puts deliverability at risk), alongside system notes for lifecycle events. Persistent issue states, including an active outbound pause, pin to the top of the feed as active platform alerts, so the center shows the outstanding issue even before anyone opens it. Every kind carries a severity (info, warning, error) and a category your preferences key on; security kinds bypass every mute. Fan-out is per-member (email, push, webhook) with an org-level digest and an org-level webhook sink that teams commonly terminate at a Slack incoming webhook, which is the path that puts incident signals in your ops channel. The taxonomy and fan-out mechanics live in the notifications hub.

​
Part 3: Steer: tenant-owned levers mid-incident

Once you know what is failing, these are your levers, narrowest first.

​
Pause and resume outbound

The pause gate is tenant-owned. Configure a spend or balance threshold under Dashboard → Billing → Alerts with action_on_hit: pause_outbound (or the harder block_outbound) before an incident, and tripping it, or editing or deleting the alert afterwards, is yours to do; the reset path is POST /api/v1/billing/alerts/reset-outbound, the same route the dashboard’s clear button calls. While a balance-driven pause holds, paid channels reject with 402 SENDING_PAUSED at admission (nothing is queued behind the gate, so treat 402 as terminal and resend after resume), while the $0 bring-your-own-credential channels (WhatsApp, Messenger, Telegram, and the rest of the exemption) keep sending. Recovery is one poll: after you resolve the cause, re-read GET /api/v1/billing/balance until outbound_paused is false. The full cause-by-cause loop is outbound pause and the resume propagation loop; the gate and the exemption are the outbound pause gate and free-channel exemption.

​
Reroute with sender pools

If specific senders or routes are the problem, rotate them out without touching your send code: edit the pool’s membership so degraded numbers leave the rotation, or let the pool’s strategy spread volume across the healthy remainder. The pool’s preview panel shows each member’s deliverability health tier and flags degraded members queued for auto-rotation, the same signal the rotation scheduler acts on. A sender belongs to one pool, so stocking a fresh pool with clean inventory is always unambiguous. The lifecycle is sender pools.

​
Shift traffic at the service level

Where your traffic enters through a messaging service, the messaging services console is the surface that decides how that service’s traffic is configured; shift new traffic onto the healthy configuration there while the incident runs, then restore afterwards.

​
Arm cascade fallback for new sends

For a degradation you can see coming, arm fallback chains on new sends: set an org cascade policy, or stamp a per-send cascade field, so a terminal failure on the primary channel escalates to the next channel automatically instead of landing in your retry queue. Each hop re-enters the canonical send pipeline, so opt-out, consent, and quiet-hours gates re-apply per hop; a recipient opted out of SMS is not rescued into SMS.

​
Emergency stop: only where it truly applies

The emergency stop is a compliance control, not an incident switch. It halts every outbound dispatch path for your organization (Verify/OTP and email excepted) and exists for the moment one more send costs more than halting everything: a suspected breach, a runaway campaign, a litigator or carrier exposure you need frozen now. A degraded carrier is not that moment; the narrower levers above pause or reroute the affected traffic while your healthy paths keep delivering. Reach for the stop only when the blast radius is the whole organization.

​
Part 4: Recover and report

​
Read the recovery signals

Recovery has the same shape as detection, in reverse:
  • Voice: Troubleshoot on the affected trunk returns status: "registered" with a non-null latency, and the fleet health snapshot stops ranking it first.
  • Messaging: the operator’s health score climbs back through its bands on GET /api/v1/messages/route-quality/scores, and suspended routes return per the circuit breaker runbook.
  • Billing: outbound_paused reads false on the balance endpoint after you clear the cause.
  • Platform: Settings → Status moves the incident to Monitoring, then Resolved.

​
Reconcile from your exports

For the post-incident accounting, export the incident window from your own records: the CDR and usage export for the voice side (CDR, usage export, and billing reconciliation) and the delivery log for messaging (delivery logs). Two counts matter:
  1. Fallback legs. Every cascaded send carries its message_group_id; group the delivery log by it to count primaries that needed a hop and to see which fallback channel carried the rescue. On the voice side, the CDR shows the fallback leg a dispatch walked when the primary trunk was unregistered.
  2. Refused sends. A pause rejects at admission rather than queueuing, so the sends that returned 402 SENDING_PAUSED or 403 ORG_COMPLIANCE_EMERGENCY_STOP during the window are a finite, enumerable set you can resend after resume, not a silent loss to hunt for.

​
Draft the customer note from tenant-owned data only

Your customer note quotes two sources and nothing else: your own traffic outcomes (delivery rates and delays from your delivery log, in your words) and the shareable status URL. Copy it with the Copy link action on Settings → Platform status; the page it opens carries no tenant data, which is what makes it safe to hand to customers, and it renders under your brand when you run an active custom domain. The org-scoped history in Settings is an internal projection of which declared windows touched your channels; it is your reading material, not a customer-facing document. Never splice the status URL by hand from an address-bar string, and never present your org-scoped view as the platform’s own declaration. The sharing mechanics live in the org-scoped status model and the platform status guide.

​
See also