Skip to main content

​
How Orbit processes work asynchronously

Most API requests on Devotel Orbit complete in one request/response cycle: you send a message, you get a 202. But a meaningful slice of platform work must survive a process restart — a billing alert that retried until the provider recovers, a contact import that outlives the API pod that accepted it, a voicemail transcription that retries for a day. Orbit runs that durable work on BullMQ queues backed by Redis, with the API producing jobs and a dedicated worker service consuming them. This page explains the producer/consumer split, the central queue registry that keeps both sides honest, how failed jobs age and get counted, and how operators inspect the whole surface from the admin health panel.

​
The three-layer async model

Work on Orbit resolves into one of three shapes, and the queue backbone covers the third one:
  1. Immediate-response paths. A request that can finish inside the call — a synchronous SMS dispatch, a number lookup, a balance read — never touches a queue. The handler completes, responds, and is done.
  2. Retriable-in-call paths. Some operations retry inside the request or handler before they give up. The Resend email helper retries a 5xx or 429 with backoff several times within the call, and only escalates to a queue when every in-call attempt fails.
  3. Durable queued work. Anything that must survive a process restart, run on a schedule, or outlive the request lifecycle — imports, retry follow-ups, delivery-receipt reconciliation, scheduled maintenance sweeps — gets enqueued to Redis and consumed by a worker, often hours or days later.
The division matters operationally: a job enqueued to Redis keeps its full payload even if the API pod that accepted it (or the worker pod that picks it up) restarts mid-flight.

​
The central queue registry

Every queue the platform produces to or consumes from is declared once, in a central registry the API, the worker service, and the admin health panel all share. Two bindings come as a pair:
  • PLATFORM_QUEUE_NAMES — an exhaustive list of the 15 queue names currently in use (webhook-delivery, webhook-dlq, contact-imports, campaign-drip-steps, dlr-retry, email-retry, migration-imports, sandbox-events, and the rest). Entries that need an unusual producer or consumer carry a comment that names both sides.
  • PLATFORM_QUEUE_CONCURRENCY — a parallel map recording the worker-side parallelism each consumer is constructed with. BullMQ never persists a worker’s concurrency into Redis, so a producer-side Queue object cannot read it back; keeping the value mirrored in the registry is the only way the health panel can show “backlog + drain rate” together. A compile-time satisfies binding forces the map to cover every declared name, so a registry entry without a concurrency value fails the build.
Both are exported from a single module, and the inventory they produce drives the queue-health endpoint automatically — a queue declared in the registry shows up in the admin panel with no extra wiring.

​
Producer/consumer split across services

The API and the worker service share the queue names, but each touches a different BullMQ primitive.
  • The API produces. Producer code constructs a Queue(name, { connection, defaultJobOptions }) and adds jobs with queue.add(name, payload, opts). A producer can optionally set defaultJobOptions — retention ages for completed/failed jobs, attempt caps — at construction time. Its role ends once the job is persisted.
  • The worker consumes. Consumer code constructs new Worker(name, processor, { concurrency }) and registers a processor callback that BullMQ invokes per job. Webhook delivery, contact imports, delivery-receipt retries, and every scheduled sweep on the platform live behind a worker in the webhook-worker service.
That service carries roughly 250 scheduler-*.ts worker files — one per concern — grouped into recognizable buckets:
  • Contact-center (ACD). Agent idle/pause timers, wrap-up transitions, callback recovery, queue SLA aging.
  • Agent memory and AI. Memory consolidation, re-embedding, turn-audit retention, trait refresh.
  • Anomaly and integrity. Event-volume anomaly scans, billing reconciliation, audit-chain (Merkle) integrity.
  • Retention TTLs. Time-bounded cleanup jobs that age rows out of hot tables (PII purge, media planes, webhook-event retention).
  • Retry queues. Email retry, delivery-receipt retry, number-release retry, voicemail-transcription retry, and the migration-import pipeline.
The prefix convention keeps one worker file per concern instead of a monolith, and the registry keeps the producer and consumer comments for each queue in one place so “who produces? who consumes?” is answered without a grep.

​
Failure semantics — what “failed” actually means

BullMQ moves a job to its failed set any time the processor throws. How long that failure stays visible — and what a queue-health badge says about it — depends on two deliberate behaviors.
  • Retention ages differ per queue. The producer and consumer each set a removeOnFail option, so a completed job can be kept for hours and a failed one for days. contact-imports, for instance, retains failures for seven days, so a Monday import error is still inspectable on Friday — but it isn’t “new”. Email-retry and several other queues behave the same way: failures are deliberately inspectable, not scrubbed on success.
  • The admin panel counts a 24-hour window, not the whole set. If GET /admin/health/queues turned a queue amber whenever any failure exists in the whole retained set, a single stale error would pin the queue in warning for a week. Health counting therefore pages the failed set newest-first and stops as soon as it crosses the 24-hour cutoff — only failures inside the last 24 hours count toward the badge. Older retained failures remain open in the inspector, but they don’t keep the queue flagged.
That distinction — how long a failure is kept versus what counts as recent — is the difference between an alert you can act on and a badge you train yourself to ignore.

​
Operator visibility

Operators read queue health from the admin Platform Health panel, behind the /admin/health route family:
  • Snapshot polling (GET /admin/health/queues). One per-queue row: waiting/active/completed/failed/delayed/paused counts plus the registry-recorded worker concurrency, so a backlog and its drain rate read side by side. The windowed failure count comes from the 24-hour page-until-cutoff sweep described above.
  • Live stream (GET /admin/health/queues/stream). A Server-Sent-Events feed that refreshes every five seconds — the operator console uses this instead of polling, so a drain-rate problem shows up mid-incident rather than at the next poll.
  • Per-job drill-down (GET /admin/health/queues/:name/jobs). Paginated job browsing by state; a list view truncates payloads and stack traces, and a per-job route returns the full payload when a failing job needs real diagnosis.
  • Bulk lifecycle actions. Retry, remove, pause/resume, drain, clean, or obliterate endpoints let an operator take corrective action from the panel; every action is audit-logged.
All of these endpoints only accept queue names from the registry — an unknown name is a 404 rather than an empty result — so the panel cannot drift away from the platform vocabulary as queues are added.

​
End-to-end example: email-retry

The email-retry flow shows the full loop — a terminal failure enqueued cleanly, retried on a long schedule, and still visible to the operator.
  1. Producer enqueues. A call site (invite, billing alert, the like) calls the Resend helper. The helper retries several times in-call. On a terminal failure — a 4xx the in-call retry refuses to re-attempt, or a sustained 5xx/network outage past the in-call budget — the producer constructs a Queue('email-retry', { removeOnComplete: { count: 500, age: 1 day }, removeOnFail: { count: 5000, age: 7 days } }) and adds a job carrying the full envelope (from/to/subject/html plus an idempotency key) with its own attempt counter, delayed five minutes. A deterministic jobId built from the idempotency key dedups concurrent enqueues.
  2. Consumer re-runs on a long schedule. The webhook-worker’s consumer constructs new Worker('email-retry', processor, { concurrency: 4 }) and re-attempts the send on a 5-minute → 30-minute → 2-hour → 12-hour → 24-hour schedule, driven by its own attempt counter (BullMQ’s internal retry budget stays at one — the schedule is bespoke, not automatic).
  3. Exhaustion is observable. When the schedule runs out, the worker writes an email_delivery_failed audit row and emits a Sentry capture, so the operator can investigate an email no customer ever received. The job lands in the failed set, retained for seven days — but it only counts toward the queue’s 24-hour “recent failures” window on day one.
The same shape covers the other investigative queues: a bounded in-call retry, a durable queue, a long consumer-side schedule, and a retained-forever failed set that only ages the badge for 24 hours.

​
What this is not

  • Not webhook delivery semantics. The webhook-delivery queue below is one queue on this backbone, but the delivery contract your endpoint sees — retries, ordering, idempotency keys, and what happens when your endpoint flaps — is covered by webhook delivery semantics. This page is about the job model, not the wire.
  • Not call queues. The CCaaS agent-queue model (ring strategy, SLA aging, callback recovery) is documented in voice queues. The ACD-named schedulers on this backbone maintain those queues; they are not how a caller gets routed.
  • Not the operational sentinels. The non-delivery statuses (test_sent, suppressed, deleted, complaint, unknown) that analytics must exclude are covered in operational sentinels. Sentinel statuses are a status-vocabulary question, not a queue-mechanics one.
The queue registry’s remaining entries (webhook-dlq, sandbox-events, and the rest) follow the same backbone — only the domain payload and the schedule change.

​
See also