> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Incident-day operations: failover drills, status surfaces, and tenant notifications

> One operator runbook for a carrier or region incident: rehearse SIP trunk and message cascade failover before it happens, read the diagnostics, status, and notification surfaces while it runs, steer traffic with tenant-owned controls, and reconcile afterwards from your own exports.

# Incident-day operations: failover drills, status surfaces, and tenant notifications

Orbit ships the controls an operator needs across a carrier or region
incident: failover you can rehearse, diagnostics that name the failing
step, org-scoped status views, notification fan-out, and tenant-owned
levers to pause, reroute, and recover traffic. Those controls are
documented on their own pages; this guide walks them in the order an
operator uses them: rehearse, detect, steer, recover and report.

Everything on this page is tenant-owned. You operate your trunks,
routes, pools, and pauses; Orbit operates the platform. Nothing here
promises platform availability; for the platform's own declarations,
read [Settings Status](https://docs.orbit.devotel.io/guides/settings-status-console.md) and the
[platform status page](https://docs.orbit.devotel.io/guides/status-page.md).

## Part 1: Rehearse failover before you need it

A failover path you have never watched walk is a failover path you do
not trust at 2 a.m. Rehearse both sides, voice and messaging, against a
non-production organization or a staging SBC.

### The SIP trunk failover drill

The mechanics are documented in
[SIP trunk connection: health checks and failover](https://docs.orbit.devotel.io/guides/sip-trunk-connection.md);
this is the drill built on them.

1. Create two outgoing trunks against your staging equipment: trunk A
   (primary) and trunk B (fallback).
2. Set `failoverTrunkId` on trunk A, pointing at trunk B. An update
   that would create a loop is rejected with a 422, so chains stay
   walkable. An incoming trunk is a valid fallback target too: its
   first allowed IP is your own already-validated SBC address.
3. Baseline both trunks. Run **Test trunk**
   (`POST /api/v1/voice/sip-trunks/{id}/test`) on each and expect
   `status: "registered"` with a non-null latency. The same probe runs
   continuously in the background, so the trunk row also carries
   `lastRegisteredAt` and `lastRegistrationError`.
4. Kill trunk A on purpose: stop the staging registrar, or point trunk
   A's `host` at an endpoint that will never answer a `REGISTER`. Run
   **Troubleshoot** on A and expect the `register` step to read
   `failed` (definitive rejection) or `pending` (no answer within the
   probe budget), with every earlier step `ok`. The fleet health
   snapshot (`GET /api/v1/voice/sip-trunks/health`) now ranks A
   unregistered, needs-attention first.
5. Place a test call through your normal outbound path. A dispatch
   that finds the primary trunk unregistered walks the failover chain
   first, then falls back to the platform default, keeping the call
   down instead of dropping it.
6. Verify the CDR shows the fallback leg on trunk B, then restore
   trunk A and confirm it re-registers before you call the drill done.

> [!NOTE]
>
> Outbound PSTN egress always stays on Orbit's wholesale path; your
> trunks connect your equipment to Orbit and never carry the platform's
> outbound exit. The drill above rehearses delivery into and out of your
> own trunks, exactly the surface a carrier or region incident touches.

### The messaging cascade drill

The messaging-side equivalent is a cascade group: a primary send plus
every fallback hop, bound to one `message_group_id`
([Cascade failover policy and logical cascade groups](https://docs.orbit.devotel.io/concepts/message-cascade-groups.md)).

1. On a sandbox key (`dv_test_sk_`), send with a typed cascade chain,
   for example a primary channel with `"cascade": { "channels": ["sms"] }`.
2. Force the primary leg to fail using the sandbox magic numbers
   ([Sandbox magic numbers playbook](https://docs.orbit.devotel.io/guides/sandbox-magic-numbers-playbook.md)):
   a recipient whose trailing digit is `3` terminates `undelivered`
   and a trailing `4` terminates `failed`, replaying the same webhook
   sequence a real carrier produces.
3. Read the group back with
   `GET /api/v1/messages/cascade/groups/<group_id>`. The drill passes
   when the response shows the primary leg terminal and a second leg
   carrying `fallback_of` (the primary's id) and `fallback_reason`
   (for example `primary_rcs_undelivered`), with `leg_count` matching
   the hops that fired. That is the chain walking, end to end, with no
   carrier involved.

The sandbox short-circuits before any provider is touched and bills
nothing, so this drill is safe to run as often as you like. The same
rehearsal works against an org-level cascade policy
(`PUT /api/v1/messages/cascade/policy`) when you want every send to
escalate through the same chain.

## Part 2: Detect: which surface shows what

During a real incident, four surfaces answer four different questions.
Read them in this order.

| Surface | Question it answers | Where |
| - | - | - |
| Trunk health and diagnostics | Is my SIP path up, and which step broke? | Voice → SIP trunks → Health, and **Troubleshoot** |
| Route quality and circuit breaker | Which destination operators are degrading, and is enforcement running? | Messages → Tools → Route quality |
| Settings → Status | Has the platform declared an incident that maps to my channels? | `/settings/status` |
| Notifications | What does the platform need to tell my team right now? | The bell, `/notifications`, and your webhook sink |

### Trunk diagnostics

The diagnostics panel's ordered steps (`resolve`, `credentials`,
`provision`, `gateway`, `register`) name the phase that broke, so a
carrier incident reads differently from a misconfiguration: a registrar
that was healthy an hour ago and now returns `register` `failed` or
`pending` across every trunk pointed at it is a carrier story, while a
`resolve` failure is usually your host field. Incoming trunks have no
REGISTER to place; they track accepted-traffic freshness instead
(`registered` within 5 minutes, `trying` within an hour, else `idle`)
and surface rejections on `lastCallVerdict` with the exact reason. The
field-by-field reading lives in
[SIP trunk health and connectivity diagnostics](https://docs.orbit.devotel.io/voice/sip-trunk-health-and-diagnostics.md).

### Route quality and the circuit breaker

For SMS and MMS, each destination operator carries a 0–100 health
score folded into a band: healthy at 85 and above, **degraded** from 60
to 84, poor from 35 to 59, critical below. A route sitting in the
degraded band with a climbing `latency_p95_ms` is the early-warning
shape of a carrier incident; a collapsed `delivery_rate` with a high
`ack_no_receipt_rate` is the grey-route shape instead. Your thresholds
decide what counts as a breach, and if you opted in, the circuit
breaker auto-suspends a sustained-failing route and fails traffic over
until it recovers. The full workflow is the
[route-quality circuit breaker runbook](https://docs.orbit.devotel.io/guides/route-quality-circuit-breaker-runbook.md).

### The org status console

**Settings → Status** maps declared platform incidents to the channels
your organization actually uses. Its incident states follow the
published progression (Investigating, Identified, Monitoring,
Resolved), and severity rolls up worst-first across overlapping
records. A channel touched by an active incident is presented at that
incident's severity; that is what a marked channel looks like on this
surface. Conversely, an incident that does not map to any of your
channels never appears in your org history, and the absence of an
incident is not proof your channel is healthy: tenant-level delivery
failures (carrier rejection, filtered content, insufficient balance)
degrade while platform availability reads 100%. The reading order is
in [Settings Status](https://docs.orbit.devotel.io/guides/settings-status-console.md); the projection
model behind it is
[the org-scoped status model](https://docs.orbit.devotel.io/concepts/org-scoped-status-model.md).

### Notifications

During an incident the kinds that fire are mostly **alerts** (a wallet
that hit zero and paused outbound, a spend threshold that tripped) and
**anomalies** (a webhook endpoint auto-disabled after repeated
failures, a DNS regression that puts deliverability at risk), alongside
**system notes** for lifecycle events. Persistent issue states,
including an active outbound pause, pin to the top of the feed as
active platform alerts, so the center shows the outstanding issue even
before anyone opens it. Every kind carries a severity (`info`,
`warning`, `error`) and a category your preferences key on; security
kinds bypass every mute. Fan-out is per-member (email, push, webhook)
with an org-level digest and an org-level webhook sink that teams
commonly terminate at a Slack incoming webhook, which is the path that
puts incident signals in your ops channel. The taxonomy and fan-out
mechanics live in the [notifications hub](https://docs.orbit.devotel.io/notifications/overview.md).

## Part 3: Steer: tenant-owned levers mid-incident

Once you know what is failing, these are your levers, narrowest first.

### Pause and resume outbound

The pause gate is tenant-owned. Configure a spend or balance threshold
under **Dashboard → Billing → Alerts** with `action_on_hit:
pause_outbound` (or the harder `block_outbound`) before an incident,
and tripping it, or editing or deleting the alert afterwards, is yours
to do; the reset path is
`POST /api/v1/billing/alerts/reset-outbound`, the same route the
dashboard's clear button calls. While a balance-driven pause holds,
paid channels reject with `402 SENDING_PAUSED` at admission (nothing is
queued behind the gate, so treat 402 as terminal and resend after
resume), while the \$0 bring-your-own-credential channels (WhatsApp,
Messenger, Telegram, and the rest of the exemption) keep sending.
Recovery is one poll: after you resolve the cause, re-read
`GET /api/v1/billing/balance` until `outbound_paused` is `false`. The
full cause-by-cause loop is
[outbound pause and the resume propagation loop](https://docs.orbit.devotel.io/concepts/outbound-pause-recovery.md);
the gate and the exemption are
[the outbound pause gate and free-channel exemption](https://docs.orbit.devotel.io/concepts/outbound-pause-and-free-channel-exemption.md).

### Reroute with sender pools

If specific senders or routes are the problem, rotate them out without
touching your send code: edit the pool's membership so degraded numbers
leave the rotation, or let the pool's strategy spread volume across the
healthy remainder. The pool's preview panel shows each member's
deliverability health tier and flags degraded members queued for
auto-rotation, the same signal the rotation scheduler acts on. A sender
belongs to one pool, so stocking a fresh pool with clean inventory is
always unambiguous. The lifecycle is
[sender pools](https://docs.orbit.devotel.io/guides/sender-pools.md).

### Shift traffic at the service level

Where your traffic enters through a messaging service, the
[messaging services console](https://docs.orbit.devotel.io/guides/messaging-services-console.md) is the
surface that decides how that service's traffic is configured; shift
new traffic onto the healthy configuration there while the incident
runs, then restore afterwards.

### Arm cascade fallback for new sends

For a degradation you can see coming, arm fallback chains on new sends:
set an org cascade policy, or stamp a per-send `cascade` field, so a
terminal failure on the primary channel escalates to the next channel
automatically instead of landing in your retry queue. Each hop
re-enters the canonical send pipeline, so opt-out, consent, and
quiet-hours gates re-apply per hop; a recipient opted out of SMS is not
rescued into SMS.

### Emergency stop: only where it truly applies

The [emergency stop](https://docs.orbit.devotel.io/compliance/emergency-stop.md) is a compliance
control, not an incident switch. It halts every outbound dispatch path
for your organization (Verify/OTP and email excepted) and exists for
the moment one more send costs more than halting everything: a
suspected breach, a runaway campaign, a litigator or carrier exposure
you need frozen now. A degraded carrier is not that moment; the
narrower levers above pause or reroute the affected traffic while your
healthy paths keep delivering. Reach for the stop only when the blast
radius is the whole organization.

## Part 4: Recover and report

### Read the recovery signals

Recovery has the same shape as detection, in reverse:

* **Voice:** **Troubleshoot** on the affected trunk returns
  `status: "registered"` with a non-null latency, and the fleet health
  snapshot stops ranking it first.
* **Messaging:** the operator's health score climbs back through its
  bands on `GET /api/v1/messages/route-quality/scores`, and suspended
  routes return per the circuit breaker runbook.
* **Billing:** `outbound_paused` reads `false` on the balance endpoint
  after you clear the cause.
* **Platform:** Settings → Status moves the incident to Monitoring,
  then Resolved.

### Reconcile from your exports

For the post-incident accounting, export the incident window from your
own records: the CDR and usage export for the voice side
([CDR, usage export, and billing reconciliation](https://docs.orbit.devotel.io/guides/cdr-usage-export-billing-reconciliation.md))
and the delivery log for messaging
([delivery logs](https://docs.orbit.devotel.io/guides/developer-delivery-logs.md)). Two counts matter:

1. **Fallback legs.** Every cascaded send carries its
   `message_group_id`; group the delivery log by it to count primaries
   that needed a hop and to see which fallback channel carried the
   rescue. On the voice side, the CDR shows the fallback leg a dispatch
   walked when the primary trunk was unregistered.
2. **Refused sends.** A pause rejects at admission rather than
   queueuing, so the sends that returned `402 SENDING_PAUSED` or
   `403 ORG_COMPLIANCE_EMERGENCY_STOP` during the window are a
   finite, enumerable set you can resend after resume, not a silent
   loss to hunt for.

### Draft the customer note from tenant-owned data only

Your customer note quotes two sources and nothing else: your own
traffic outcomes (delivery rates and delays from your delivery log, in
your words) and the shareable status URL. Copy it with the **Copy
link** action on **Settings → Platform status**; the page it opens
carries no tenant data, which is what makes it safe to hand to
customers, and it renders under your brand when you run an active
custom domain. The org-scoped history in Settings is an internal
projection of which declared windows touched your channels; it is your
reading material, not a customer-facing document. Never splice the
status URL by hand from an address-bar string, and never present your
org-scoped view as the platform's own declaration. The sharing
mechanics live in
[the org-scoped status model](https://docs.orbit.devotel.io/concepts/org-scoped-status-model.md) and
[the platform status guide](https://docs.orbit.devotel.io/guides/status-page.md).

## See also

* [SIP trunk connection: health checks and failover](https://docs.orbit.devotel.io/guides/sip-trunk-connection.md)
* [SIP trunk health and connectivity diagnostics](https://docs.orbit.devotel.io/voice/sip-trunk-health-and-diagnostics.md)
* [Cascade failover policy and logical cascade groups](https://docs.orbit.devotel.io/concepts/message-cascade-groups.md)
* [Sandbox magic numbers playbook](https://docs.orbit.devotel.io/guides/sandbox-magic-numbers-playbook.md)
* [Route-quality circuit breaker runbook](https://docs.orbit.devotel.io/guides/route-quality-circuit-breaker-runbook.md)
* [Settings Status console](https://docs.orbit.devotel.io/guides/settings-status-console.md)
* [The org-scoped status model](https://docs.orbit.devotel.io/concepts/org-scoped-status-model.md)
* [Notifications hub](https://docs.orbit.devotel.io/notifications/overview.md)
* [Outbound pause gate and free-channel exemption](https://docs.orbit.devotel.io/concepts/outbound-pause-and-free-channel-exemption.md)
* [Outbound pause and the resume propagation loop](https://docs.orbit.devotel.io/concepts/outbound-pause-recovery.md)
* [Sender pools](https://docs.orbit.devotel.io/guides/sender-pools.md)
* [Emergency stop](https://docs.orbit.devotel.io/compliance/emergency-stop.md)
* [CDR, usage export, and billing reconciliation](https://docs.orbit.devotel.io/guides/cdr-usage-export-billing-reconciliation.md)
* [Delivery logs](https://docs.orbit.devotel.io/guides/developer-delivery-logs.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.