Skip to main content

​
Troubleshooting: failed webhook deliveries, retries, DLQ, and replay

Your dashboard shows failed deliveries, your endpoint returned 5xx or timed out, or events simply stopped arriving. This page walks you through the delivery lifecycle, from the first failure to a deliberate replay, and tells you which recovery tool fits which failure. Signature failures are a different class — if your endpoint rejects every delivery with a 401 or a signature error, fix verification first with Troubleshooting: signature failures and come back here once the endpoint accepts requests.

​
Symptom map

Match what you see to the underlying cause before reaching for a recovery tool: Also check the reliability roll-up (GET /api/v1/webhooks/reliability) when you cannot tell whether one endpoint or the whole tenant is affected — it summarizes success rate and the oldest pending delivery per endpoint over a 24-hour window.

​
Retry cadence: when a row lands in the DLQ

Orbit retries a failed delivery automatically: up to 9 retries on an exponential backoff with a 30-second base delay that doubles with each attempt, plus up to 20% jitter — roughly 4.3 hours from the first failure to the final attempt, per the FAQ and the Webhook security page’s delivery contract. Two implications:
  • Do not manually requeue a row that is still retrying. A row that shows failed on an attempt is not exhausted until it appears in the DLQ. Wait for the scheduler to finish its 4.3-hour window before treating the event as lost.
  • Fix the endpoint before the scheduler finishes. If a deploy broke the consumer, a hotfix redeployed within the window lets one of the remaining retries succeed on its own — no manual action needed.
After the final attempt fails, the delivery moves to the dead-letter queue with status dlq and remains retrievable.

​
Step 1 — Read the delivery attempts

List the attempts for the endpoint and filter to the failure:
Cursor-paginate through the window you care about (from_date / to_date bounds it) and pull the individual delivery (GET /api/v1/webhooks/{id}/deliveries/{deliveryId}) to read the exact payload Orbit sent and the HTTP status your endpoint returned. That pair answers what failed: a 404 means the route moved, a 502/503 means the app crashed or its upstream did, a timeout means the handler ran past the endpoint’s timeout_seconds. For the request shape and filters, see List webhook deliveries.

​
Step 2 — DLQ listing and single-delivery requeue

List the dead-letter queue, optionally scoped to one endpoint:
When to requeue a single delivery — with POST /api/v1/webhooks/dlq/{deliveryId}/requeue: one row, still meaningful, after you confirmed the endpoint is healthy again. Requeueing resets the delivery to pending and clears its attempt counter, so the retry scheduler picks it up with a fresh nine-attempt budget. When NOT to requeue — when the endpoint is still failing. A requeue into an endpoint that still returns 5xx just burns the fresh budget and puts the row back in the DLQ nine attempts later. Fix the consumer first, then requeue. If hundreds of rows are queued behind the same outage, a single requeue per row is the wrong tool — that is what bulk replay is for.

​
Step 3 — Replay: single delivery and bulk

Requeue and replay are not interchangeable:
  • Requeue moves a DLQ row back into automatic retries. Use it when the failure was transient and you want the normal scheduler to handle it.
  • Replay re-dispatches the stored payload through the standard dispatch pipeline, on demand. Unlike requeue, replay works on succeeded deliveries too — which is exactly why it can double-process (see idempotency below).
Single-delivery replay — POST /api/v1/webhooks/{id}/deliveries/{deliveryId}/replay — when you need that one event re-sent now rather than on the retry cadence: the customer is on the phone, or the event gates a downstream workflow. Bulk replay — POST /api/v1/webhooks/{id}/replay-range — when an outage caused a window of failures and the DLQ rows are all attributable to it. The request takes a from/to window and an optional eventTypeFilter, returns 202 with a job_id, and runs as a background job. Poll it with GET /api/v1/webhooks/{id}/replay-range/{jobId} until it reports done or failed:
Constraints to plan around:
  • One bulk replay per endpoint. A second start while a job is queued or running returns 409 with error code BULK_REPLAY_IN_PROGRESS. Wait for the in-flight job to finish rather than looping on the 409.
  • Keep the window tight. Replay re-sends every stored delivery in the window, including ones that succeeded. A window wider than the outage replays events your consumer already processed — see idempotency below.
When not to replay a row: when your consumer still is not idempotent and the row may already have been processed (a succeeded row in the window, or a row on which the handler reached its side effect before the ack failed). Fix the consumer’s dedup first — the next section — then replay.

​
Idempotency: replay-safe consumers

Replay re-delivers the original payload with the same event id. A consumer that already persisted the event must recognize the re-delivery and no-op, because replay deliberately re-sends events that may have succeeded. The durable-consumer pattern — persist the seen event id, ack fast, process through a queue — makes every recovery tool above safe to use. Build that first, then replay is a routine operation instead of a double-processing risk: see the Build a durable webhook consumer guide.

​
What to send support

If deliveries keep failing after the endpoint returns 2xx consistently, or a bulk replay job reports failed, collect the identifiers before you write in — each one removes a round-trip:
  • Tenant id (Settings → Organization)
  • Webhook endpoint id (the endpoint from the dashboard URL or GET /api/v1/webhooks)
  • Delivery id from the failed row (from the deliveries list or DLQ)
  • For bulk replay problems, the job id from the 202 response, plus the from/to window you replayed
  • The time range of the failures you saw, in UTC

​
See also