Skip to main content

​
Event consumer groups: managed offsets

For high-volume event pipelines, tracking your own cursor against the raw event feed is the wrong tool. GET /events is stateless cursor paging — your client carries the cursor, and if it loses that position it re-reads or skips. GET /events/stream is a live SSE tail with Last-Event-ID resumes. Neither gives you a server-side committed position, replay control, or lag visibility. Consumer groups add that managed layer over the same ordered per-tenant event stream. The model is deliberately Kafka-shaped: a poll reads from a durable server-side offset, and your consumer commits that offset back only after it has processed the batch. That gives you at-least-once delivery, replay/backfill, and a lag report, without running a broker. The field reference is the Events API.

​
1. The three semantic concepts

The whole model is built from three small pieces:
  • Consumer-group name — an identifier you pick (1–128 chars of letters, digits, ._:-, starting alphanumeric; e.g. dlr-warehouse, siem-sync). All offsets are tracked per group, so each independent consumer of your stream uses its own group and advances its own cursor.
  • Offset — a position in the stream. The server stores one committed offset per group and everything reads relative to it. Offsets compare numerically, not lexically, so ordering is always correct.
  • Verbs — GET /events/consume (poll a batch), POST /events/consume/commit (advance the group’s offset), POST /events/consume/seek (reset it, the only operation allowed to move backwards), and GET /events/consumer-groups (list groups with lag).

​
2. At-least-once delivery

A poll (GET /events/consume?group=...&limit=...) reads the batch after the group’s committed offset but does not advance it. You commit (POST /events/consume/commit) only after your side has durably processed — written to the warehouse, handed to the queue, stored the checkpoint — the batch. Until you commit, the batch’s events remain available to the group. That gives at-least-once delivery: a crash or restart before commit re-delivers the same batch on the next poll. The platform does not deduplicate for you — consumers dedup on each event’s id (a stable stream id the poll attaches to each event’s envelope). Idempotency in your consumer is the only dedup layer. The commit itself is idempotent and monotonic: submitting the same offset, or an offset already behind the committed position, never rewinds the cursor. The response’s applied flag tells you whether the write took. Each poll returns the shape you need to drive this loop: committed_offset (the position the batch was read after), next_offset (the offset to commit after processing the batch — equals committed_offset when the batch was empty), count, and has_more (true when the batch filled your limit).

​
3. The cursor model

Offsets are opaque: <ms>-<seq> stream ids, or the literal 0 for the origin of the retained window. Treat them as tokens you pass back verbatim — do not construct or parse them on your side. The committed offset survives disconnects. It lives server-side per group, so a consumer that crashes and restarts — or that you redeploy — picks up exactly where it last committed. A group that has never committed starts at 0, the beginning of the retained window. A seek to end on an empty stream resolves to 0 too (== “from now”).

​
4. Isolation

Consumer groups are per tenant. Your offsets live under your own tenant event channel, and there is no cross-tenant iterator — a group never sees another tenant’s events, and another tenant’s groups never see yours. That makes the feature safe to share across all tenants without coordination. Within your own tenant, independence is by group name: two consumers that should each see the full stream use two distinct group names; one logical consumer scaled across worker replicas uses one group name all replicas share, so the committed cursor is a single durable position.

​
5. Failure shapes to plan for

The failure modes are deliberately loud and bounded:
  • Consumer-group overflow — a tenant may hold at most 100 consumer groups. The limit exists so a client that mints a fresh group name per poll cannot grow the offset store without bound. The commit/seek write for a new group past the limit returns 409 with code CONSUMER_GROUP_LIMIT; writes from an existing group are unaffected. Reuse a small set of stable group names.
  • Seek ahead of head — numeric offset comparison means a seek to an explicit offset that is past the stream head simply parks the group past everything currently retained; only future events then poll through. Seeking to a malformed offset is rejected at validation with a 422 before any write.
  • Stale offsets when a tenant re-attaches — offsets are durable for 30 days, refreshed on every commit/seek. After a fully dormant window a group can lose its committed offset; its next poll then falls back to the beginning of the retained window (never to skipping events).
  • Duplicated delivery — at-least-once means redelivery after any crash-before-commit. Dedup on the event id.
GET /events/consumer-groups surfaces each group’s committed offset and approximate lag (retained events waiting after the committed offset) so you can alarm on a consumer falling behind. Lag is reported with a lag_capped flag once a group’s backlog exceeds the report cap, so the list call stays cheap even for far-behind groups.

​
6. A worked example

Poll a batch for a group, then commit the returned next_offset once you have processed it:
To backfill over the retained window, seek then consume:
Send exactly one of to or offset; sending both or neither is a 422.

​
7. Contrast with the other event surfaces

Orbit’s event plane has three egress shapes, and the trade-off is worth naming before you pick one:
  • CDP streaming destinations are fan-out: every event is produced onto your Kafka topic or Kinesis stream, and your downstream consumers manage the bus-side offset. Pick this when your event system of record is already a broker.
  • Event sources inbound is the reverse direction — Orbit consumes your Kafka topic in. That model shares the filter and envelope vocabulary with the sink side, but it is an inbound subscription, not a managed offset over Orbit’s own stream.
  • Consumer groups fit when you want Kafka-style pull semantics without standing up a bus: your pipeline polls and commits Orbit-side, and the only state you carry is the group name.
The event taxonomy itself — which event types exist and what each envelope carries — is defined in the firehose and schema registry surfaces; consumer groups are a cursor layer over that taxonomy, not a new event model.

​
See also

Events API

The consume / commit / seek / consumer-groups endpoint field reference.

CDP streaming destinations

Kafka and Kinesis fan-out when your consumers own a bus.

Event sources inbound model

The reverse direction — your Kafka topic consumed into Orbit.

Webhook fan-out and event sinks

The push-based sibling over the same event taxonomy.