Skip to main content

​
Configure route-quality thresholds and run the circuit breaker

This runbook walks the full workflow: read your per-operator route-health scores, set the thresholds that decide what counts as a breach, check whether a route is breaching right now, and opt in to the circuit breaker that auto-suspends a sustained-failing route until it recovers. Work through it once and the route-quality surface runs on your own settings from then on. Everything on this page is tenant-owned: the thresholds, the monitoring window, the probe schedule, and whether enforcement runs are your settings — they apply to your traffic only. Nothing here changes how messages terminate; route quality measures the SMS and MMS routes you already use.

​
What a route-health score is

For SMS and MMS, each destination operator you send to gets a transparent health score from 0 to 100, built from signals your own message history already carries: delivery conversion, delivery-receipt (DLR) latency percentiles, and the acknowledged-without-receipt share. The score folds into a band — healthy, degraded, poor, or critical — and a route earns a grey-route / SIM-box flag when its sink and fake-accept tells corroborate on a material sample. Email and WhatsApp messages carry no route score; they deliver over provider APIs, not carrier routes. For the model, the weighting, and why these are the tells that matter, read the concept page: Outbound route quality and circuit breakers. The rest of this page is the workflow.

​
Read the scores

GET /api/v1/messages/route-quality/scores returns every destination operator your traffic reached in the window, scored. The window runs 1–90 days; the default is 7.
Your API key needs the messages:read scope. The same table renders in the dashboard at Messages → Tools → Route quality, where a window picker covers the last 1, 7, 14, 30, or 90 days.
Read each row left to right — the same columns the dashboard table shows:
  • mccmnc / mcc — the destination operator (mobile country + network code of the receiving handset) and its country. Scores are per operator, so two operators in one country get separate rows.
  • terminal — the sample: terminal-status messages backing the row. A route needs a material sample before its score means anything; below the scorer’s minimum sample a row reads a neutral 100 rather than flagging you on a handful of sends.
  • health_score + band — 0–100 composite (100 is a clean route) and its band: healthy ≥ 85, degraded ≥ 60, poor ≥ 35, critical below that. In the dashboard the score renders as a bar tinted by the band.
  • delivery_rate — delivered ÷ terminal, in percent. A sustained collapse against real volume is the classic grey-route tell: the carrier accepts the traffic and sinks it.
  • latency_p50_ms / latency_p95_ms / latency_p99_ms — delivery-receipt latency from submit to handset, in milliseconds. The p95 is the early-warning meter: extra intermediary hops inflate the tail before the delivery rate moves. null means no timed deliveries in the window.
  • ack_no_receipt_rate — the share of terminal traffic the route accepted without ever returning a delivery receipt, in percent. High values are the SIM-box “fake accept” pattern.
  • grey_route_suspected — the corroborated flag; see Reading the suspect flag below.
  • factors — the penalty breakdown: which signals subtracted from 100, by how much, with a plain-text detail each. Use it to answer “why is this route at 33?” without re-deriving the math.
The response also carries your current thresholds, the alerts evaluation against them, and the circuit-breaker snapshot — so the dashboard renders the table and the breach banner in one call, without notifying anyone.

​
Reading the suspect flag

grey_route_suspected: true means the two corroborating tells crossed their threshold together on a material sample: the route sinks deliveries (low delivery_rate) and fakes accepts (high ack_no_receipt_rate). Either tell alone doesn’t flag — a route that merely delivers poorly, or one that merely under-reports receipts, isn’t flagged. When both fire together you have the strongest pattern a single sender can see, and in the dashboard the row badges Grey route instead of the band label. Treat a flagged route as a candidate for a failover conversation with your provider, not as proof by itself — the concept page covers the tells and their limits. Pick window length to match the question: 1–7 days for “is this route bad right now,” 30–90 days to separate a sustained degradation from one noisy day. The circuit breaker works off whichever window you check it with, so set your window deliberately rather than accepting the default.

​
Set your thresholds

Thresholds define what counts as a breach for your tenant. Read them first, then write the full settings object back — PUT replaces the settings, so always include the fields you mean to keep.
The write takes the messages:write scope and the owner, admin, or developer role — the same gate the dashboard’s threshold form enforces. Each threshold:
  • min_health_score (0–100, default 70) — breach when a route’s score drops below this. Raise it for OTP and other latency-critical traffic; lower it for bulk marketing where a degraded route is tolerable.
  • max_latency_p95_ms (default 30000) — breach when the p95 delivery-receipt latency exceeds this. Thirty seconds catches clear intermediary inflation without paging on normal carrier jitter; tighten toward 10–15s if your use case is time-sensitive.
  • alert_on_grey_route (default true) — breach whenever a route carries the corroborated grey-route / SIM-box flag. Leave this on; it is the earliest discrete signal in the set.
  • min_volume (default 50) — the terminal-message sample a route needs before it can breach at all. A route below the minimum is too small to judge, so it pages nobody. Set it relative to your send pattern: hourly cron traffic wants a lower floor than a bulk sender.
  • severity (warning or critical, default warning) — the severity stamped on each breach, which rides through to the notification. Use critical only when a breached route should wake someone.
In the dashboard the same fields live under Alert thresholds on the Route quality page; Save thresholds writes the identical settings object.

​
Loopback probes

Scores are passive — they measure traffic you already send, so a route you send to in bursts goes dark between bursts. A loopback probe closes that gap: you schedule periodic test sends to a named operator, and the probe travels the same send pipeline as your real traffic, keeping the route’s score fed between bursts. Add entries to loopback_probes in the same PUT. Each probe is an operator mccmnc, a friendly label, a cadence_minutes between 5 and 1440, and an enabled flag:
Probes count as sends — they move through the standard send path and are billed like any other message — so point them at routes that matter and keep cadences reasonable. The dashboard lists the active schedule under Scheduled loopback probes.

​
Check whether a route is breaching now

POST /api/v1/messages/route-quality/alert-check evaluates the current window against your thresholds and pages you for each breach — the manual “did anything just go red” action, and the shape a scheduled alert run would use.
Read the response as:
  • alerts.breached — did anything cross your thresholds in this window. false means clean; the call still returns the full scores table for context.
  • alerts.breaches[] — one breach per route, picking the most severe reason (grey_route over low_health_score over high_latency) so one degraded operator pages once. Each entry names the measured value, the threshold it crossed (null for the flag-based grey-route reason), the severity, the volume behind it, and a plain-text detail.
  • notifications_dispatched — the breaches that actually paged. Dispatch dedupes per operator, reason, and severity on a cooldown window, so re-running the check against a persisting breach reports it in alerts.breaches without spamming your notification bell — a breach that was already paged shows up in breaches but not again in notifications_dispatched.
In the dashboard this is the Check & notify button at the top of the Route quality page, evaluated over the window you have selected. A 200 with breached: false and an empty notifications_dispatched is the clean-state result.

​
Opt in to the circuit breaker

The advisory layer pages you; the circuit breaker acts on sustained breaches. It is opt-in — enabled defaults to false, and until you turn it on a route is never suspended and the alert-check workflow above is the whole story. The breaker is a per-route state machine. A route starts closed (traffic flows normally). After trip_after_consecutive_breaches consecutive failing checks, it opens and the route is auto-suspended. After cooldown_minutes, it moves to half-open and deliberately lets checks through to test recovery; half_open_recovery_checks consecutive clean checks return it to closed, while a single breach during the probe re-opens it and restarts the cooldown. The full contract — including where the failover lands — is in the circuit-breaker concept. Enable it by writing the circuit_breaker block alongside your thresholds:
The knobs:
  • enabled — master switch, default off. Off means advisory-only forever.
  • trip_after_consecutive_breaches (1–20, default 3) — how many consecutive breaching checks trip a route open. The sustained-breach gate: one noisy window never takes a route offline. Lower it for OTP-class traffic; raise it where flapping costs more than degrading.
  • cooldown_minutes (1–1440, default 30) — how long an open route sits before the half-open probe starts. Match it to how quickly a route can plausibly heal; a carrier-side sink rarely recovers in five minutes.
  • half_open_recovery_checks (1–20, default 2) — consecutive clean checks during the probe that confirm recovery.
  • failover_channel — where suspended traffic diverts: whatsapp, rcs, viber, or telegram, or null for no failover. The honest reason for the restriction is that failover should land on a rich-messaging channel over a provider API, not on another degradable carrier route. The recorded channel feeds the same fallback machinery described in Per-message cascade policy on smart-send and Fallback chains; recipients unreachable on the failover channel don’t move.

​
Run the breaker

Enabling the policy arms it; each circuit check advances the state machine. Call it directly:
Each call returns the new circuit_states, the derived suspended_routes list (what is actually suspended right now), and the transitions this check made — with one notification per transition wired to the Route quality page. When the breaker is disabled, the call returns enabled: false and changes nothing, so checking it is always safe. While a route is open, the read-only GET /scores response surfaces the same suspended list, and the decision each send consults — suspended or not, and which failover channel — is part of the persisted state. A tripped route stays suspended until checks prove it clean again; to stand a route back up sooner, call the check once the route genuinely recovers rather than flipping enabled off.

​
Worked example: a degrading operator, start to recovery

Watch one operator, 23415, degrade over a 7-day window. The dashboard row follows the API at every step. 1. Healthy. GET /scores?days=7 shows health_score: 97, band: "healthy", delivery 96%, p95 under five seconds. The dashboard row shows a full-green bar and a Healthy badge. Nothing to do. 2. Degrading. Mid-week the route’s delivery rate slides to 60% and the p95 inflates past your 20s ceiling:
The row now reads health_score: 51, band: "poor", and the dashboard tints the bar amber. Below your min_health_score of 75 but not yet flagged as grey — this is the low_health_score band, and if enforcement is on the consecutive-breach counter has started. 3. Flagged and suspended. Two more sliding-window sends confirm the pattern: the accepted-without-receipt share climbs and deliveries keep sinking, so grey_route_suspected flips to true and the row earns the Grey route badge. On the third consecutive breaching check, the breaker trips:
The response carries transitions: [{ "type": "tripped", "mccmnc": "23415" }] and suspended_routes: ["23415"]; your notification bell gets the suspension with the configured failover channel named. New traffic to 23415 now fails over per your chain instead of walking into the sink. 4. Probing. After 30 minutes of cooldown the route moves to half-open and the next checks deliberately pass — you see "state": "half_open" in circuit_states while the score recovers. This is deliberately noisy-looking: traffic is flowing on a route you suspended, by design, to test it. 5. Recovered. Two consecutive clean checks close the circuit:
suspended_routes empties, the row returns to a Healthy badge, and traffic routes normally. If the route had breached during the probe instead, the transition would read reopened, the cooldown would restart, and the failover would hold — which is exactly the behavior you want from a route that heals partially and relapses.

​
Tenant-owned framing

Every control in this runbook is scoped to your tenant and owned by you. The thresholds decide what a breach means for your traffic; the probe schedule spends your sends on the routes you care about; the breaker’s trip tolerance, cooldown, and failover channel encode your appetite for automation; and enabled decides whether any of it acts at all. A suspension applies to your traffic only, and scores are computed from your message history alone — no cross-tenant signal feeds them. A tenant that has never configured any of it reads the documented defaults and an empty suspended list, never an error.

​
See also