Skip to main content

​
Operator observability: what to look at when

Orbit exposes four operator surfaces for watching your traffic — delivery logs, live request/event streams, analytics dashboards and Insights, and the audit trail — plus three ways to feed data into your own systems (webhooks, polling exports, scheduled reports). This page frames what each surface is for, so you stop opening a dashboard to find a single message, or paging over an ephemeral event stream when you need a durable record.

​
The surfaces at a glance

​
Delivery logs vs dashboards vs audit — three different questions

Operators reach for a surface by habit; the honest distinction is the granularity of the question. Moment-level (a single thing). The delivery log and live tail resolve individual messages and requests. A correlated lookup on a msg_… id in the delivery log is the canonical way to answer “did this exact send arrive” — no aggregate can tell you that, and an aggregate blurred by replica lag can disagree with it for up to three seconds. Trend-level (a shape over time). The Analytics API and Insights API are read-replica-backed deliberately: they are cheap to call for dash-rendering precisely because they accept a 1–3 second lag behind the primary. Treat either as the answer to “what does the last N days look like,” and not as the answer to “what happened to message X two seconds ago.” Delivery outcome aggregates also exclude operational sentinels (test sends, suppressions, deletes) from their denominators — mirror that discipline in your own reporting or your delivery-rate math drifts. Durable record (keep it, prove it). The Events firehose is explicitly not durable — it is a bounded, in-memory replay buffer (~5000 events, ~15-minute idle TTL) that degrades open when the real-time store is unavailable. Anything you would build a system of record, reconciliation job, or compliance archive off of must come from webhooks (durable, retried, signed delivery to your endpoint) or from exports — the vCon conversation export signs each export at emit time, and the conversation archive produces bulk CSV/JSON for filtered slices. The read-facing Events API page itself says this; treat it as a convenience for live dashboards, not an audit log.

​
Push, poll, or schedule — feeding your own systems

Three transport shapes exist; pick by how often and who owns the storage.
  • Webhooks — push, real-time, durable. The preferred feed for anything with a retention requirement. Orbit delivers the event envelope to your endpoint with signature headers, retries, and a dead-letter queue; your system owns the record from then on. Use webhooks when you need a durable event history, when you drive a downstream workflow (Sync to CRM, kick a ticket, update a contact), or when the 1–3 s replica lag on dashboards matters.
  • Polling the APIs — on demand, latest-window. The Events GET and per-message GET /messages/:id calls are right for ad-hoc reconciliation, or for operator UIs that poll the current state of a message after an operator delete/cancel (no webhook fires for those flows). Do not build a polling loop on the events buffer as your system of record — evicted ids are gone.
  • Scheduled exports — batch, emailed. For the daily/weekly trend rollup that a stakeholder reads, POST /analytics/scheduled-reports registers a recurring report and Orbit renders and emails it. Prefer this over building a polling job that re-derives the same aggregate. Creating or editing a schedule requires an owner or admin role because the payload carries per-tenant PII (top contacts, spend).

​
Alerting — where thresholds live

Alerting is not one surface; each metric class has its own rule surface, and the right home depends on the thing being measured.
  • AI spend alerts live on the Insights budget: PUT /insights/llm-spend/budget sets the daily-cap and threshold that the by-feature endpoint reports breaches against. The overview endpoint returns cap_percentage so your wallboard can render it.
  • Queue / voice alerts live as wallboard alarm rules: each rule binds a metric (waiting, longest_wait, service_level, abandon_rate, agents_available), a comparator, and a threshold; omit queue_id for a tenant-wide rule. The wallboard raises while the breach persists, so wire only the thresholds you intend to supervise.
  • QA review workload alerting lives on the QA workload surface: due-date windows and per-evaluator caps surface the backlog before it builds. The dashboard renders it; agents see only their own scores.
  • Webhook-health alerting is on you: watch for dead-letter entries (webhooks troubleshooting) and alert on your endpoint’s failure rate.

​
Putting it together

A healthy operator posture uses all four surfaces on purpose:
  1. Dashboards (Analytics, Insights) answer trend questions, accepting the read-replica lag that makes them cheap.
  2. Moment-lookup surfaces (delivery log, live tail) answer “what exactly happened to this thing.”
  3. Durable history (webhooks, vCon export, archive) backs anything retention or compliance touches.
  4. Alerts are placed on the right rule surface (Insights budget, wallboard alarm rules, QA workload), not rolled ad-hoc.
The failure modes this map exists to prevent: treating the events buffer as an audit log; polling dashboards as a delivery mechanism and missing the replica lag; treating sentinel statuses as delivery outcomes; and piling every alert onto one surface where it drowns.

​
See also