> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Queue inspector: watch every BullMQ queue, drill into jobs, and recover in bulk

> Operators-only console that spans every BullMQ queue on the platform — live depth monitoring, per-queue job drill-in, bulk retry/remove, and audit-logged recovery actions.

# Queue inspector

The queue inspector is the operators-only console for the platform's BullMQ backbone. Where the older DLQ pages each cover one queue family, the inspector spans **every queue the platform produces to** — webhook delivery, contact imports, campaign steps, delivery-receipt retries, email retries, migration imports, sandbox events, and the rest — from a single grid.

Use it to watch queue depths live during an incident, drill into a specific queue's jobs by state, and take corrective action — retry, remove, pause, drain, clean — without shell access. Every write action is audit-logged with the actor's user id and IP.

## 1. Queue inspector vs. the DLQ pages

Pick the surface by scope:

* **Webhook DLQ** — one queue (`webhook-dlq`): outbound webhook deliveries that exhausted their retries, with per-delivery retry and bulk replay. Covered in [Webhook endpoint operations](/guides/webhook-endpoint-operations).
* **CDP DLQ** — failed fan-out to CDP destination subscriptions, with single retry and per-destination replay. Covered in [CDP event debugger and DLQ](/guides/cdp-event-debugger-and-dlq).
* **Queue inspector** — the cross-queue view. It supersedes the webhook-only DLQ page for operational work: the same retry/remove workflow, applied to **any** BullMQ queue, plus live depth monitoring and full lifecycle controls (pause, resume, drain, clean, obliterate) that neither DLQ page carries.

Reach for the inspector when the problem is not "one webhook endpoint is down" but "which queue is backing up, and what do I do about it."

## 2. Live queue-depth monitoring

The overview is a card grid — one card per queue — that streams fresh counts over Server-Sent Events. A status line above the grid shows **Live (SSE)** while the stream is connected, or **Polling fallback** when the stream is unavailable and the page refreshes the snapshot every 10 seconds instead. Either way you read the same numbers; SSE delivers them mid-incident rather than at the next poll.

Each card shows six counters per queue:

| Counter | State     | What it tells you                            |
| ------- | --------- | -------------------------------------------- |
| `w`     | Waiting   | Backlog not yet picked up                    |
| `a`     | Active    | Jobs running right now                       |
| `d`     | Delayed   | Scheduled retries waiting on their timer     |
| `f`     | Failed    | Terminal failures, retained per queue policy |
| `p`     | Paused    | Jobs held by a queue pause                   |
| `c`     | Completed | Recently finished work (retained briefly)    |

Each card also shows the queue's **worker concurrency** — the drain rate the backlog numbers must be read against. A waiting count of 2,000 on a queue with concurrency 4 is a different problem from 2,000 waiting on concurrency 64.

Card badges follow a simple rule: any failed count flags the queue **failed**, a waiting count over 100 flags **waiting**, otherwise **healthy**. One caveat to keep in mind: the failed counter reflects the retained failed set — queues keep failures for days by design, so a nonzero `f` does not always mean a current incident. Drill in and check timestamps before acting.

## 3. Drill into a queue

Select a card to open the job inspector for that queue. The inspector pages through jobs **one state at a time** — pick the state (failed, waiting, active, delayed, paused, completed) from the selector, then move through the list 25 jobs at a time with **Prev** / **Next**. Prev replays the exact cursor path you came in on, so paging backwards lands on the same window you saw before.

Each row shows the job id, job name, attempts made, enqueue timestamp, the failed reason and first stacktrace line for failed jobs, and a bounded payload preview. Two things are deliberately truncated in the list to keep it scannable: payload bodies (500 characters) and stack traces (first line only).

To see the full record, either:

* Click **Details** on the row — the detail drawer fetches the untruncated payload, the full stacktrace array, the current state, and the job's options.
* Paste a known id into **Find job by id…** and submit — this jumps straight to the drawer for that job without paging, which is the fastest path when you already have an id from a log line or a Sentry issue.

## 4. Bulk retry and bulk remove

Within the current state view, check jobs individually or use **Select all** for the page, then run one of the two bulk actions. Both take an explicit confirmation dialog that states the count, the queue, and the state you're acting on — never a bare "Confirm."

* **Retry** re-queues the selected jobs from the **failed** or **delayed** state. Retry is state-agnostic underneath, but the action is only offered for those two states — retrying a waiting or active job is meaningless, and the console refuses to suggest it. The dialog's confirmation button repeats the action and count — `Retry 42 jobs` — and the operation is recorded in the audit log with the state you were actually viewing, so the audit record and the operator's intent cannot diverge.
* **Remove** permanently deletes the selected jobs from the current state view. Irreversible; same confirmation shape.

Both actions accept up to 500 selected job ids per call. For a state-wide sweep beyond 500 jobs, use **Clean** (below) instead of paging through selections.

## 5. Queue lifecycle controls

The lifecycle card per queue carries five controls, with friction scaled to blast radius:

| Control        | Effect                                                                                                                         | Gate                      |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------ | ------------------------- |
| **Pause**      | Workers stop picking up new jobs; active jobs finish. Reversible, no data loss.                                                | Single click              |
| **Resume**     | Restarts pickup after a pause.                                                                                                 | Single click              |
| **Drain**      | Removes every **waiting** job — and delayed jobs too, if you check that box. Active, completed, and failed jobs are untouched. | Confirm dialog            |
| **Clean**      | Removes up to 1,000 jobs in one chosen state older than a grace window (immediately, 1 hour, 24 hours, or 7 days).             | Confirm dialog            |
| **Obliterate** | Irreversibly destroys the queue and every job in it. Fails if the queue still has active jobs.                                 | Type the exact queue name |

Pause is the right first move for a runaway producer: it freezes new intake while you triage, and resume restores normal flow with no data loss. Drain and Clean are the backlog-removal tools. Obliterate exists for rebuilding a queue from scratch and is the strongest gate in the console — the confirmation input only enables the button when the text matches the queue name exactly.

## 6. Walkthrough: recover a poison queue

The recurring incident this console is built for: one job shape fails deterministically, and its retries flood a queue.

1. **Spot the backlog.** The overview shows the queue's `w` counter climbing while `a` stays pinned — workers are burning capacity on retries that all land in `f`. Select the queue.
2. **Pause intake.** Open **Queue lifecycle → Pause**. Workers finish their active jobs and stop picking up new ones; the flood stops growing. Note the failed counter on the card no longer increments once active work drains.
3. **Diagnose one failure.** In the job inspector set the state to `failed`, open **Details** on the newest row, and read the full failed reason and stacktrace. If the failure shares a payload shape (same job name, same input), you have the confirmation you need.
4. **Fix the cause.** Ship the worker fix, or correct the upstream producer. **Resume** the queue when the cause is gone — new jobs process normally.
5. **Clear the poison backlog.** Set the state to `failed`, **Select all**, and **Retry** the jobs that are valid under the fix — the dialog confirms `Retry N jobs`. Jobs that were malformed by the old producer (not just failed under it) are not worth retrying: select them and **Remove**, or use **Clean** on the failed state with the grace window that isolates them.
6. **Verify in the audit log.** Every action above — the pause, the resume, each bulk retry and remove, any clean — lands in the audit log with your user id, IP, the queue name, and the affected counts. Close the incident by checking the audit entries match what you intended; the [audit log guide](/guides/audit-log) covers reading the ledger.

## 7. Audit trail and access

The inspector is an operators-only surface, scoped to authorised operations staff — it is not exposed to tenant users. Every mutating action it can take (bulk retry, bulk remove, pause, resume, drain, clean, obliterate) writes an audit-log entry with the actor's user id and IP, the queue, the state acted on, and the per-job counts. Access reviews and incident post-mortems stay grounded in the audit log, not in recollection of who clicked what.

## See also

* [How Orbit processes work asynchronously](/concepts/async-processing-model) — the BullMQ backbone, queue registry, and failure semantics the inspector reads
* [Webhook endpoint operations](/guides/webhook-endpoint-operations) — the webhook-only DLQ workflow for per-endpoint delivery failures
* [CDP event debugger and DLQ](/guides/cdp-event-debugger-and-dlq) — the destination-scoped replay workflow for CDP subscriptions
* [Audit log](/guides/audit-log) — reading the ledger every inspector action writes to