Skip to main content

​
Queue inspector

The queue inspector is the operators-only console for the platform’s BullMQ backbone. Where the older DLQ pages each cover one queue family, the inspector spans every queue the platform produces to — webhook delivery, contact imports, campaign steps, delivery-receipt retries, email retries, migration imports, sandbox events, and the rest — from a single grid. Use it to watch queue depths live during an incident, drill into a specific queue’s jobs by state, and take corrective action — retry, remove, pause, drain, clean — without shell access. Every write action is audit-logged with the actor’s user id and IP.

​
1. Queue inspector vs. the DLQ pages

Pick the surface by scope:
  • Webhook DLQ — one queue (webhook-dlq): outbound webhook deliveries that exhausted their retries, with per-delivery retry and bulk replay. Covered in Webhook endpoint operations.
  • CDP DLQ — failed fan-out to CDP destination subscriptions, with single retry and per-destination replay. Covered in CDP event debugger and DLQ.
  • Queue inspector — the cross-queue view. It supersedes the webhook-only DLQ page for operational work: the same retry/remove workflow, applied to any BullMQ queue, plus live depth monitoring and full lifecycle controls (pause, resume, drain, clean, obliterate) that neither DLQ page carries.
Reach for the inspector when the problem is not “one webhook endpoint is down” but “which queue is backing up, and what do I do about it.”

​
2. Live queue-depth monitoring

The overview is a card grid — one card per queue — that streams fresh counts over Server-Sent Events. A status line above the grid shows Live (SSE) while the stream is connected, or Polling fallback when the stream is unavailable and the page refreshes the snapshot every 10 seconds instead. Either way you read the same numbers; SSE delivers them mid-incident rather than at the next poll. Each card shows six counters per queue: Each card also shows the queue’s worker concurrency — the drain rate the backlog numbers must be read against. A waiting count of 2,000 on a queue with concurrency 4 is a different problem from 2,000 waiting on concurrency 64. Card badges follow a simple rule: any failed count flags the queue failed, a waiting count over 100 flags waiting, otherwise healthy. One caveat to keep in mind: the failed counter reflects the retained failed set — queues keep failures for days by design, so a nonzero f does not always mean a current incident. Drill in and check timestamps before acting.

​
3. Drill into a queue

Select a card to open the job inspector for that queue. The inspector pages through jobs one state at a time — pick the state (failed, waiting, active, delayed, paused, completed) from the selector, then move through the list 25 jobs at a time with Prev / Next. Prev replays the exact cursor path you came in on, so paging backwards lands on the same window you saw before. Each row shows the job id, job name, attempts made, enqueue timestamp, the failed reason and first stacktrace line for failed jobs, and a bounded payload preview. Two things are deliberately truncated in the list to keep it scannable: payload bodies (500 characters) and stack traces (first line only). To see the full record, either:
  • Click Details on the row — the detail drawer fetches the untruncated payload, the full stacktrace array, the current state, and the job’s options.
  • Paste a known id into Find job by id… and submit — this jumps straight to the drawer for that job without paging, which is the fastest path when you already have an id from a log line or a Sentry issue.

​
4. Bulk retry and bulk remove

Within the current state view, check jobs individually or use Select all for the page, then run one of the two bulk actions. Both take an explicit confirmation dialog that states the count, the queue, and the state you’re acting on — never a bare “Confirm.”
  • Retry re-queues the selected jobs from the failed or delayed state. Retry is state-agnostic underneath, but the action is only offered for those two states — retrying a waiting or active job is meaningless, and the console refuses to suggest it. The dialog’s confirmation button repeats the action and count — Retry 42 jobs — and the operation is recorded in the audit log with the state you were actually viewing, so the audit record and the operator’s intent cannot diverge.
  • Remove permanently deletes the selected jobs from the current state view. Irreversible; same confirmation shape.
Both actions accept up to 500 selected job ids per call. For a state-wide sweep beyond 500 jobs, use Clean (below) instead of paging through selections.

​
5. Queue lifecycle controls

The lifecycle card per queue carries five controls, with friction scaled to blast radius: Pause is the right first move for a runaway producer: it freezes new intake while you triage, and resume restores normal flow with no data loss. Drain and Clean are the backlog-removal tools. Obliterate exists for rebuilding a queue from scratch and is the strongest gate in the console — the confirmation input only enables the button when the text matches the queue name exactly.

​
6. Walkthrough: recover a poison queue

The recurring incident this console is built for: one job shape fails deterministically, and its retries flood a queue.
  1. Spot the backlog. The overview shows the queue’s w counter climbing while a stays pinned — workers are burning capacity on retries that all land in f. Select the queue.
  2. Pause intake. Open Queue lifecycle → Pause. Workers finish their active jobs and stop picking up new ones; the flood stops growing. Note the failed counter on the card no longer increments once active work drains.
  3. Diagnose one failure. In the job inspector set the state to failed, open Details on the newest row, and read the full failed reason and stacktrace. If the failure shares a payload shape (same job name, same input), you have the confirmation you need.
  4. Fix the cause. Ship the worker fix, or correct the upstream producer. Resume the queue when the cause is gone — new jobs process normally.
  5. Clear the poison backlog. Set the state to failed, Select all, and Retry the jobs that are valid under the fix — the dialog confirms Retry N jobs. Jobs that were malformed by the old producer (not just failed under it) are not worth retrying: select them and Remove, or use Clean on the failed state with the grace window that isolates them.
  6. Verify in the audit log. Every action above — the pause, the resume, each bulk retry and remove, any clean — lands in the audit log with your user id, IP, the queue name, and the affected counts. Close the incident by checking the audit entries match what you intended; the audit log guide covers reading the ledger.

​
7. Audit trail and access

The inspector is an operators-only surface, scoped to authorised operations staff — it is not exposed to tenant users. Every mutating action it can take (bulk retry, bulk remove, pause, resume, drain, clean, obliterate) writes an audit-log entry with the actor’s user id and IP, the queue, the state acted on, and the per-job counts. Access reviews and incident post-mortems stay grounded in the audit log, not in recollection of who clicked what.

​
See also