Troubleshooting: webhook ordering and fan-out

Your state mirror shows a message stuck at sent even though delivered came in minutes ago, a status jumps backwards (a completed call flips to in_progress), or the same event processed twice applied a stale value over a fresh one. Almost every “webhooks arrive out of order” report lands in one of three buckets: your handler applies events in arrival order, your fan-out across endpoints lets different consumers disagree about “latest”, or your supersede key is the wrong column. This page maps each symptom, then gives you the ordering fix the at-least-once model requires. Duplicates — the same event_id delivered more than once — are a distinct failure class with its own page: start with Troubleshooting: duplicate webhook events and consumer-side dedup and come back here once dedup is covered. The delivery-semantics model this page operationalizes is on Webhook delivery semantics.

1. What the delivery model promises

Orbit’s webhook contract is at-least-once, with no ordering guarantee. Events are emitted in event-time order per endpoint, but network delivery can reorder them: a slow first attempt lands after a later event’s first attempt, a retry is interleaved with newer events by construction, and a DLQ replay arrives whenever you fire it. Three consequences define the ordering surface:

Arrival order is meaningless. The only sortable key on an envelope is created_at — the event-time timestamp Orbit stamped when the event was recorded, identical on every retry and replay of that event.
There is no global ordering across event types. A message.sent and a message.delivered for the same message, or an contact.updated and a message.delivered, arrive in whatever sequence the network produces. Supersede per entity, per attribute — never assume type A precedes type B.
Fan-out reorders by endpoint. When two endpoints subscribe to the same event type — a dashboard endpoint and a warehouse endpoint — they receive deliveries on their own schedules. Each endpoint sees its own order; a consumer reading across endpoints sees a merged, re-ordered stream.

2. The symptom table

Match the wrong-order symptom you see to its cause before changing code:

Symptom	What it usually means	Where to confirm
`message.delivered` applied, then `message.sent` flips the status back	Handler applies by arrival order; the retry of `sent` landed after the first `delivered` attempt	Consumer logs — the two event ids arrive out of `created_at` order
Entity shows a stale latest value (status, name, opt-in flag)	Supersede keyed on arrival, or on `event_id` sequence instead of `created_at`	Compare persisted value vs. the event with max `created_at`
Same `event_id` re-processed and applied a stale value over a fresh one	Dedup insert succeeded, but the side effect re-ran on the duplicate with an older payload	Dedup table hit vs. the value the duplicate wrote
Fan-out reordering across endpoints — dashboard shows one status, warehouse shows another	Each endpoint is its own ordering stream; consumers disagree on “latest”	`GET /api/v1/webhooks` — two endpoints both subscribed to the event type
Two simultaneous POSTs, both writing — last one in wins arbitrarily	No per-entity lock or optimistic update; concurrency undoes `created_at` ordering	Consumer logs — same entity id, two interleaved writes
DLQ replay reinserts old state after a recovered endpoint caught up	Replayed events carry their original `created_at`; your apply logic treats replay as fresh	DLQ requeue log vs. the value the replay overwrote

Each row expands below with the remedy it points to.

3. Remedies — the ordering-correct consumer

The fix layers on top of dedup. Dedup decides whether an event is new; ordering decides what a new event changes. Four remedies, in the order you usually compose them: Remedy 1 — Dedupe on body.id first. Persist the envelope id (evt_...) with a unique constraint and ack duplicates 200. This stops a retry or replay from re-running your apply logic with an older payload that would supersede a fresher value. The dedup machinery is on the duplicate-events page; ordering assumes it is already in place. Remedy 2 — Apply-by-created_at supersede per entity. When you mirror entity state into your own datastore, never INSERT rows for status transitions and read the latest by arrival — UPDATE the entity row guarded by the event-time timestamp:

UPDATE messages
SET    status = $2,
       last_event_id = $3,
       updated_at = $4
WHERE  id = $1
  AND  (updated_at IS NULL OR updated_at < $4);

An event whose created_at is older than what you already stored becomes a no-op — exactly the semantics the model promises tolerating. Where multiple attributes update independently (status vs. a metadata field), guard per attribute or keep per-attribute updated_at columns so a late status does not clobber a fresh name. Remedy 3 — Idempotent state mirror. Treat the receiving handler as a projection, not a ledger. The handler persists the dedup id, enqueues the event, and lets a worker apply Remedy 2 against an upsert table of current state. Crashes mid-apply retry the worker job without new deliveries; the destination table converges to the event with max created_at per entity — the same invariant whether the event arrived once, twice, or after a replay. Remedy 4 — Queue-based consumer with ordering per entity key. Where concurrency breaks your ordering (two parallel POSTs racing one entity), route work through a queue keyed on the entity — message.id, call.id, conversation.id. One in-flight job per entity serializes applies for that entity so the created_at supersede is a single-thread comparison instead of a data race; different entities still process in parallel. A keyed queue also makes bulk replay safe — the window re-sends duplicates, dedup absorbs them, and the remaining events land in entity-serial order. Two consequences fall out of the pattern:

The only correct sort key is the envelope created_at — never arrival order, never the delivery-attempt sequence, and (for non-idempotent types) never the lexicographic event_id.
Replay is ordering-neutral by construction. A DLQ requeue or bulk replay re-sends stored events with their original created_at; your supersede guard drops them if newer state already landed.

4. Where it fires

Ordering sits at the intersection of three pages — the concept model, the delivery/recovery loop, and dedup — and one reference:

Webhook delivery semantics — the at-least-once / unordered contract this page operationalizes.
Troubleshooting: failed webhook deliveries, retries, DLQ, and replay — the failure-replay loop that reorders re-sent events onto your endpoint.
Troubleshooting: duplicate webhook events and consumer-side dedup — the dedup half of the pair Remedy 1 assumes.
Webhook events reference — the per-event-type inventory, so you know which entity the event mutates and which created_at supersede to apply.

5. Sample fixture — a deliberately reordered pair

Two events for one message, delivered in reverse:

// Arrives SECOND (first-attempt of the earlier event lost the race)
{ "id": "evt_09f3d1",
  "type": "message.sent",
  "created_at": "2026-09-05T10:14:12.120Z",
  "data": { "id": "msg_5177", "status": "sent" } }

// Arrives FIRST (its first attempt was fast)
{ "id": "evt_09f3d2",
  "type": "message.delivered",
  "created_at": "2026-09-05T10:14:14.470Z",
  "data": { "id": "msg_5177", "status": "delivered" } }

A receiver keyed on arrival ends with status = sent — wrong. The correct apply, in pseudocode:

on_event(e):
  if seen.contains(e.id):                      # Remedy 1 — dedup on body.id
      return 200                               # duplicate: ack, no apply
  seen.insert(e.id)

  entity = queue_key(e.data.id)                # Remedy 4 — per-entity lock
  with_lock(entity):
      row = db.fetch(e.data.id)
      if row is null or e.created_at > row.updated_at:   # Remedy 2 — supersede
          db.upsert(e.data.id, status = e.data.status,
                    updated_at = e.created_at, last_event_id = e.id)
      # else: older event — no-op, still ack 200

Apply the first-arriving delivered (10:14:14.470) into an empty row; when the second-arriving sent (10:14:12.120) lands, its created_at is older than the stored updated_at, so the guard rejects it as a no-op and the row stays delivered. A replay of either event later re-hits the same guard and still no-ops.

What to send support

If you see events whose created_at order on Orbit’s side contradicts the order your receiver recorded — i.e. dedup is clean, supersede is guarded, and a stale value still landed — write in with:

The two event ids (evt_...) in the misapplied pair, plus the entity id (msg_... / call_...) they mutate.
The timestamps — each event’s created_at, your receive time, and your apply time, in UTC.
Endpoint ids if fan-out across endpoints is involved.
Tenant id (Settings → Organization) and the delivery ids from the deliveries list.

That set lets support line up the dispatch log against your apply log in one pass.

Reference

Troubleshooting: webhook ordering and fan-out — out-of-order application under at-least-once

Troubleshooting: webhook ordering and fan-out

1. What the delivery model promises

2. The symptom table

3. Remedies — the ordering-correct consumer

4. Where it fires

5. Sample fixture — a deliberately reordered pair

What to send support

See also

​Troubleshooting: webhook ordering and fan-out

​1. What the delivery model promises

​2. The symptom table

​3. Remedies — the ordering-correct consumer

​4. Where it fires

​5. Sample fixture — a deliberately reordered pair

​What to send support

​See also

Troubleshooting: webhook ordering and fan-out

1. What the delivery model promises

2. The symptom table

3. Remedies — the ordering-correct consumer

4. Where it fires

5. Sample fixture — a deliberately reordered pair

What to send support

See also