> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting: webhook ordering and fan-out — out-of-order application under at-least-once

> Fix webhooks applied in the wrong order — a later event overwriting an earlier one, fan-out arrested across several endpoints, and 'delivered before sent' — by superseding on the envelope created_at and keying idempotent upserts per entity.

# Troubleshooting: webhook ordering and fan-out

Your state mirror shows a message stuck at `sent` even though `delivered`
came in minutes ago, a status jumps backwards (a `completed` call flips to
`in_progress`), or the same event processed twice applied a stale value over
a fresh one. Almost every "webhooks arrive out of order" report lands in
one of three buckets: your handler applies events in **arrival order**, your
fan-out across endpoints lets different consumers disagree about "latest",
or your supersede key is the wrong column. This page maps each symptom, then
gives you the ordering fix the at-least-once model requires.

Duplicates — the same `event_id` delivered more than once — are a distinct
failure class with its own page: start with
[Troubleshooting: duplicate webhook events and consumer-side dedup](/troubleshooting/webhook-event-dedup)
and come back here once dedup is covered. The delivery-semantics model this
page operationalizes is on
[Webhook delivery semantics](/concepts/webhook-delivery-semantics).

## 1. What the delivery model promises

Orbit's webhook contract is **at-least-once, with no ordering guarantee**.
Events are emitted in event-time order per endpoint, but network delivery
can reorder them: a slow first attempt lands after a later event's first
attempt, a retry is interleaved with newer events by construction, and a DLQ
replay arrives whenever you fire it.

Three consequences define the ordering surface:

* **Arrival order is meaningless.** The only sortable key on an envelope is
  `created_at` — the event-time timestamp Orbit stamped when the event was
  recorded, identical on every retry and replay of that event.
* **There is no global ordering across event types.** A `message.sent` and a
  `message.delivered` for the same message, or an `contact.updated` and a
  `message.delivered`, arrive in whatever sequence the network produces.
  Supersede per entity, per attribute — never assume type A precedes type B.
* **Fan-out reorders by endpoint.** When two endpoints subscribe to the same
  event type — a dashboard endpoint and a warehouse endpoint — they receive
  deliveries on their own schedules. Each endpoint sees its own order; a
  consumer reading across endpoints sees a merged, re-ordered stream.

## 2. The symptom table

Match the wrong-order symptom you see to its cause before changing code:

| Symptom                                                                                   | What it usually means                                                                            | Where to confirm                                                         |
| ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------ |
| `message.delivered` applied, then `message.sent` flips the status back                    | Handler applies by arrival order; the retry of `sent` landed after the first `delivered` attempt | Consumer logs — the two event ids arrive out of `created_at` order       |
| Entity shows a stale latest value (status, name, opt-in flag)                             | Supersede keyed on arrival, or on `event_id` sequence instead of `created_at`                    | Compare persisted value vs. the event with max `created_at`              |
| Same `event_id` re-processed and applied a stale value over a fresh one                   | Dedup insert succeeded, but the side effect re-ran on the duplicate with an older payload        | Dedup table hit vs. the value the duplicate wrote                        |
| Fan-out reordering across endpoints — dashboard shows one status, warehouse shows another | Each endpoint is its own ordering stream; consumers disagree on "latest"                         | `GET /api/v1/webhooks` — two endpoints both subscribed to the event type |
| Two simultaneous POSTs, both writing — last one in wins arbitrarily                       | No per-entity lock or optimistic update; concurrency undoes `created_at` ordering                | Consumer logs — same entity id, two interleaved writes                   |
| DLQ replay reinserts old state after a recovered endpoint caught up                       | Replayed events carry their original `created_at`; your apply logic treats replay as fresh       | DLQ requeue log vs. the value the replay overwrote                       |

Each row expands below with the remedy it points to.

## 3. Remedies — the ordering-correct consumer

The fix layers on top of dedup. Dedup decides *whether* an event is new;
ordering decides *what* a new event changes. Four remedies, in the order you
usually compose them:

**Remedy 1 — Dedupe on `body.id` first.** Persist the envelope `id`
(`evt_...`) with a unique constraint and ack duplicates `200`. This stops a
retry or replay from re-running your apply logic with an older payload that
would supersede a fresher value. The dedup machinery is on the
[duplicate-events page](/troubleshooting/webhook-event-dedup); ordering
assumes it is already in place.

**Remedy 2 — Apply-by-`created_at` supersede per entity.** When you mirror
entity state into your own datastore, never `INSERT` rows for status
transitions and read the latest by arrival — `UPDATE` the entity row guarded
by the event-time timestamp:

```sql theme={null}
UPDATE messages
SET    status = $2,
       last_event_id = $3,
       updated_at = $4
WHERE  id = $1
  AND  (updated_at IS NULL OR updated_at < $4);
```

An event whose `created_at` is older than what you already stored becomes a
no-op — exactly the semantics the model promises tolerating. Where multiple
attributes update independently (status vs. a metadata field), guard per
attribute or keep per-attribute `updated_at` columns so a late `status`
does not clobber a fresh `name`.

**Remedy 3 — Idempotent state mirror.** Treat the receiving handler as a
projection, not a ledger. The handler persists the dedup id, enqueues the
event, and lets a worker apply Remedy 2 against an upsert table of current
state. Crashes mid-apply retry the worker job without new deliveries; the
destination table converges to the event with max `created_at` per entity —
the same invariant whether the event arrived once, twice, or after a replay.

**Remedy 4 — Queue-based consumer with ordering per entity key.** Where
concurrency breaks your ordering (two parallel POSTs racing one entity),
route work through a queue keyed on the entity — `message.id`, `call.id`,
`conversation.id`. One in-flight job per entity serializes applies for that
entity so the `created_at` supersede is a single-thread comparison instead of
a data race; different entities still process in parallel. A keyed queue
also makes bulk replay safe — the window re-sends duplicates, dedup absorbs
them, and the remaining events land in entity-serial order.

Two consequences fall out of the pattern:

* **The only correct sort key is the envelope `created_at`** — never arrival
  order, never the delivery-attempt sequence, and (for non-idempotent types)
  never the lexicographic `event_id`.
* **Replay is ordering-neutral by construction.** A DLQ requeue or bulk
  replay re-sends stored events with their original `created_at`; your
  supersede guard drops them if newer state already landed.

## 4. Where it fires

Ordering sits at the intersection of three pages — the concept model, the
delivery/recovery loop, and dedup — and one reference:

* [Webhook delivery semantics](/concepts/webhook-delivery-semantics) — the
  at-least-once / unordered contract this page operationalizes.
* [Troubleshooting: failed webhook deliveries, retries, DLQ, and replay](/troubleshooting/webhook-deliveries) —
  the failure-replay loop that reorders re-sent events onto your endpoint.
* [Troubleshooting: duplicate webhook events and consumer-side dedup](/troubleshooting/webhook-event-dedup) —
  the dedup half of the pair Remedy 1 assumes.
* [Webhook events reference](/reference/webhook-events) — the per-event-type
  inventory, so you know which entity the event mutates and which `created_at`
  supersede to apply.

## 5. Sample fixture — a deliberately reordered pair

Two events for one message, delivered in reverse:

```json theme={null}
// Arrives SECOND (first-attempt of the earlier event lost the race)
{ "id": "evt_09f3d1",
  "type": "message.sent",
  "created_at": "2026-09-05T10:14:12.120Z",
  "data": { "id": "msg_5177", "status": "sent" } }

// Arrives FIRST (its first attempt was fast)
{ "id": "evt_09f3d2",
  "type": "message.delivered",
  "created_at": "2026-09-05T10:14:14.470Z",
  "data": { "id": "msg_5177", "status": "delivered" } }
```

A receiver keyed on arrival ends with `status = sent` — wrong. The correct
apply, in pseudocode:

```text theme={null}
on_event(e):
  if seen.contains(e.id):                      # Remedy 1 — dedup on body.id
      return 200                               # duplicate: ack, no apply
  seen.insert(e.id)

  entity = queue_key(e.data.id)                # Remedy 4 — per-entity lock
  with_lock(entity):
      row = db.fetch(e.data.id)
      if row is null or e.created_at > row.updated_at:   # Remedy 2 — supersede
          db.upsert(e.data.id, status = e.data.status,
                    updated_at = e.created_at, last_event_id = e.id)
      # else: older event — no-op, still ack 200
```

Apply the first-arriving `delivered` (10:14:14.470) into an empty row; when
the second-arriving `sent` (10:14:12.120) lands, its `created_at` is older
than the stored `updated_at`, so the guard rejects it as a no-op and the row
stays `delivered`. A replay of either event later re-hits the same guard and
still no-ops.

## What to send support

If you see events whose `created_at` order on Orbit's side contradicts the
order your receiver recorded — i.e. dedup is clean, supersede is guarded,
and a stale value still landed — write in with:

* **The two event ids** (`evt_...`) in the misapplied pair, plus the
  entity id (`msg_...` / `call_...`) they mutate.
* **The timestamps** — each event's `created_at`, your receive time, and
  your apply time, in UTC.
* **Endpoint ids** if fan-out across endpoints is involved.
* **Tenant id** (Settings → Organization) and the **delivery ids** from
  the deliveries list.

That set lets support line up the dispatch log against your apply log in
one pass.

## See also

* [Webhook delivery semantics](/concepts/webhook-delivery-semantics) — the
  model page this runbook presumes.
* [Troubleshooting: duplicate webhook events and consumer-side dedup](/troubleshooting/webhook-event-dedup) —
  the at-least-once duplicate half.
* [Troubleshooting: failed webhook deliveries, retries, DLQ, and replay](/troubleshooting/webhook-deliveries) —
  the failure-replay loop.
* [Build a durable webhook consumer](/guides/webhook-consumer) — the
  worked receiver that implements the full pattern.