> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# How Orbit processes work asynchronously

> The queue and worker backbone behind durable work on Orbit — what gets enqueued instead of answered inline, how producers and consumers split across services, how failures are counted, and how operators read queue health.

# How Orbit processes work asynchronously

Most API requests on Devotel Orbit complete in one request/response cycle: you send a message, you get a `202`. But a meaningful slice of platform work must survive a process restart — a billing alert that retried until the provider recovers, a contact import that outlives the API pod that accepted it, a voicemail transcription that retries for a day. Orbit runs that durable work on **BullMQ** queues backed by Redis, with the API producing jobs and a dedicated worker service consuming them.

This page explains the producer/consumer split, the central queue registry that keeps both sides honest, how failed jobs age and get counted, and how operators inspect the whole surface from the admin health panel.

## The three-layer async model

Work on Orbit resolves into one of three shapes, and the queue backbone covers the third one:

1. **Immediate-response paths.** A request that can finish inside the call — a synchronous SMS dispatch, a number lookup, a balance read — never touches a queue. The handler completes, responds, and is done.
2. **Retriable-in-call paths.** Some operations retry *inside* the request or handler before they give up. The Resend email helper retries a 5xx or 429 with backoff several times within the call, and only escalates to a queue when every in-call attempt fails.
3. **Durable queued work.** Anything that must survive a process restart, run on a schedule, or outlive the request lifecycle — imports, retry follow-ups, delivery-receipt reconciliation, scheduled maintenance sweeps — gets enqueued to Redis and consumed by a worker, often hours or days later.

The division matters operationally: a job enqueued to Redis keeps its full payload even if the API pod that accepted it (or the worker pod that picks it up) restarts mid-flight.

## The central queue registry

Every queue the platform produces to or consumes from is declared once, in a central registry the API, the worker service, and the admin health panel all share. Two bindings come as a pair:

* **`PLATFORM_QUEUE_NAMES`** — an exhaustive list of the 15 queue names currently in use (`webhook-delivery`, `webhook-dlq`, `contact-imports`, `campaign-drip-steps`, `dlr-retry`, `email-retry`, `migration-imports`, `sandbox-events`, and the rest). Entries that need an unusual producer or consumer carry a comment that names both sides.
* **`PLATFORM_QUEUE_CONCURRENCY`** — a parallel map recording the worker-side parallelism each consumer is constructed with. BullMQ never persists a worker's concurrency into Redis, so a producer-side `Queue` object cannot read it back; keeping the value mirrored in the registry is the only way the health panel can show "backlog + drain rate" together. A compile-time `satisfies` binding forces the map to cover every declared name, so a registry entry without a concurrency value fails the build.

Both are exported from a single module, and the inventory they produce drives the queue-health endpoint automatically — a queue declared in the registry shows up in the admin panel with no extra wiring.

## Producer/consumer split across services

The API and the worker service share the queue names, but each touches a different BullMQ primitive.

* **The API produces.** Producer code constructs a `Queue(name, { connection, defaultJobOptions })` and adds jobs with `queue.add(name, payload, opts)`. A producer can optionally set `defaultJobOptions` — retention ages for completed/failed jobs, attempt caps — at construction time. Its role ends once the job is persisted.
* **The worker consumes.** Consumer code constructs `new Worker(name, processor, { concurrency })` and registers a processor callback that BullMQ invokes per job. Webhook delivery, contact imports, delivery-receipt retries, and every scheduled sweep on the platform live behind a worker in the webhook-worker service.

That service carries roughly **250 `scheduler-*.ts` worker files** — one per concern — grouped into recognizable buckets:

* **Contact-center (ACD).** Agent idle/pause timers, wrap-up transitions, callback recovery, queue SLA aging.
* **Agent memory and AI.** Memory consolidation, re-embedding, turn-audit retention, trait refresh.
* **Anomaly and integrity.** Event-volume anomaly scans, billing reconciliation, audit-chain (Merkle) integrity.
* **Retention TTLs.** Time-bounded cleanup jobs that age rows out of hot tables (PII purge, media planes, webhook-event retention).
* **Retry queues.** Email retry, delivery-receipt retry, number-release retry, voicemail-transcription retry, and the migration-import pipeline.

The prefix convention keeps one worker file per concern instead of a monolith, and the registry keeps the producer and consumer comments for each queue in one place so "who produces? who consumes?" is answered without a grep.

## Failure semantics — what "failed" actually means

BullMQ moves a job to its **failed set** any time the processor throws. How long that failure stays visible — and what a queue-health badge says about it — depends on two deliberate behaviors.

* **Retention ages differ per queue.** The producer and consumer each set a `removeOnFail` option, so a completed job can be kept for hours and a failed one for days. `contact-imports`, for instance, retains failures for seven days, so a Monday import error is still inspectable on Friday — but it isn't "new". Email-retry and several other queues behave the same way: failures are deliberately inspectable, not scrubbed on success.
* **The admin panel counts a 24-hour window, not the whole set.** If `GET /admin/health/queues` turned a queue amber whenever *any* failure exists in the whole retained set, a single stale error would pin the queue in warning for a week. Health counting therefore pages the failed set newest-first and stops as soon as it crosses the 24-hour cutoff — **only failures inside the last 24 hours count toward the badge**. Older retained failures remain open in the inspector, but they don't keep the queue flagged.

That distinction — *how long a failure is kept* versus *what counts as recent* — is the difference between an alert you can act on and a badge you train yourself to ignore.

## Operator visibility

Operators read queue health from the admin **Platform Health** panel, behind the `/admin/health` route family:

* **Snapshot polling (`GET /admin/health/queues`).** One per-queue row: waiting/active/completed/failed/delayed/paused counts plus the registry-recorded worker concurrency, so a backlog and its drain rate read side by side. The windowed failure count comes from the 24-hour page-until-cutoff sweep described above.
* **Live stream (`GET /admin/health/queues/stream`).** A Server-Sent-Events feed that refreshes every five seconds — the operator console uses this instead of polling, so a drain-rate problem shows up mid-incident rather than at the next poll.
* **Per-job drill-down (`GET /admin/health/queues/:name/jobs`).** Paginated job browsing by state; a list view truncates payloads and stack traces, and a per-job route returns the full payload when a failing job needs real diagnosis.
* **Bulk lifecycle actions.** Retry, remove, pause/resume, drain, clean, or obliterate endpoints let an operator take corrective action from the panel; every action is audit-logged.

All of these endpoints only accept queue names from the registry — an unknown name is a `404` rather than an empty result — so the panel cannot drift away from the platform vocabulary as queues are added.

## End-to-end example: `email-retry`

The email-retry flow shows the full loop — a terminal failure enqueued cleanly, retried on a long schedule, and still visible to the operator.

1. **Producer enqueues.** A call site (invite, billing alert, the like) calls the Resend helper. The helper retries several times *in-call*. On a terminal failure — a 4xx the in-call retry refuses to re-attempt, or a sustained 5xx/network outage past the in-call budget — the producer constructs a `Queue('email-retry', { removeOnComplete: { count: 500, age: 1 day }, removeOnFail: { count: 5000, age: 7 days } })` and adds a job carrying the full envelope (`from`/`to`/`subject`/`html` plus an idempotency key) with its own attempt counter, delayed five minutes. A deterministic `jobId` built from the idempotency key dedups concurrent enqueues.
2. **Consumer re-runs on a long schedule.** The webhook-worker's consumer constructs `new Worker('email-retry', processor, { concurrency: 4 })` and re-attempts the send on a 5-minute → 30-minute → 2-hour → 12-hour → 24-hour schedule, driven by its own attempt counter (BullMQ's internal retry budget stays at one — the schedule is bespoke, not automatic).
3. **Exhaustion is observable.** When the schedule runs out, the worker writes an `email_delivery_failed` audit row and emits a Sentry capture, so the operator can investigate an email no customer ever received. The job lands in the failed set, retained for seven days — but it only counts toward the queue's 24-hour "recent failures" window on day one.

The same shape covers the other investigative queues: a bounded in-call retry, a durable queue, a long consumer-side schedule, and a retained-forever failed set that only ages the badge for 24 hours.

## What this is not

* **Not webhook delivery semantics.** The `webhook-delivery` queue below is one queue on this backbone, but the *delivery contract* your endpoint sees — retries, ordering, idempotency keys, and what happens when your endpoint flaps — is covered by [webhook delivery semantics](/concepts/webhook-delivery-semantics). This page is about the job model, not the wire.
* **Not call queues.** The CCaaS agent-queue model (ring strategy, SLA aging, callback recovery) is documented in [voice queues](/guides/voice-queues). The ACD-named schedulers on this backbone *maintain* those queues; they are not how a caller gets routed.
* **Not the operational sentinels.** The non-delivery statuses (`test_sent`, `suppressed`, `deleted`, `complaint`, `unknown`) that analytics must exclude are covered in [operational sentinels](/concepts/operational-sentinels). Sentinel statuses are a status-vocabulary question, not a queue-mechanics one.

The queue registry's remaining entries (`webhook-dlq`, `sandbox-events`, and the rest) follow the same backbone — only the domain payload and the schedule change.

## See also

* [Webhook delivery semantics](/concepts/webhook-delivery-semantics) — the delivery contract on top of the `webhook-delivery` queue
* [Voice queues](/guides/voice-queues) — the CCaaS agent-queue model the ACD schedulers maintain
* [Operational sentinels](/concepts/operational-sentinels) — the non-delivery statuses that live beside the queue backbone
* [Idempotency and safe retries](/concepts/idempotency-and-safe-retries) — why dedup keys make deterministic queue ids safe
* [Send gating and quiet hours](/concepts/send-gating-and-quiet-hours) — the in-call policy gates that fire before enqueueing