Skip to main content

​
Event sources: the inbound consumer model

An event source is the inbound mirror of an event sink. Sinks stream the platform event taxonomy out of Orbit to your own Kafka topic or HTTP collector; a source consumes your own Kafka topic into Orbit. If Kafka is your event system of record, a source subscribes to your domain events — orders, payments, fraud signals, CRM updates — and ingests them through the same filter and envelope semantics a sink publishes with. This page frames that inbound model. The how-to is the event sources guide; the endpoint field reference is in the Developer API.

​
1. The two directions — sink out, source in

Orbit’s event plane has two symmetrical directions, and a source is the inbound half: The symmetry is deliberate, not cosmetic. The source’s subscription filter is the sink filter, reused verbatim — exact type, <prefix>.* glob, or * for everything — so inbound and outbound subscription semantics can never drift. The envelope a source ingests is the same { type, data } shape a sink publishes: one taxonomy, one filter vocabulary, both directions. (The outbound topology — one platform event fanning out to webhook endpoints and sinks — is covered in Webhook fan-out and event sinks.)

​
2. The config model

A source is configured by a small set of fields held per kind (today kafka is the only source kind; the model is a discriminated union so further kinds slot in without reshaping stored config). The fields split into three groups:
  • Subscription shaping — the source blob: kind (kafka), the topic to consume, the consumer_group (Kafka’s group.id for this subscription), and an optional type_path. A topic follows Kafka’s naming rule ([A-Za-z0-9._-], 1–249 chars, never . or ..); the consumer group follows the same safe charset, 1–255 chars.
  • Connection — bootstrap brokers (one host:port entry each, no scheme or path, up to 16), plus an optional SASL mechanism (plain, scram-sha-256, scram-sha-512) and username.
  • The write-only credential — the SASL password. It encrypts at rest on save and never returns from any read surface; a read exposes only a credentials_configured presence flag, so the dashboard password field always starts blank and a value is sent only when you type a new one. An explicit empty-string value removes the stored secret. This is the same credential model the sink side uses — identical split, either direction.
One subscription field deserves a second look: type_path is the dot-path into each decoded message that carries the event type. It defaults to type; set something like meta.event if your envelope nests the type. The path read never crosses the prototype chain, so a path segment like constructor resolves to nothing rather than to inherited state. PATCH on the config endpoint is a merge — send only what changes. The source.kind discriminator must equal the :kind path segment, and a broker entry carrying a scheme is rejected; full field shapes live in the Developer API reference, with the step-by-step in the event sources guide.

​
3. The dry-run consume-batch validation flow

Before any transport consumes, you validate the topic, type path, and filter against sample messages you supply. The Test consume batch action (dashboard, or POST .../consume-batch directly) runs a deterministic consume engine over your pasted batch and returns a per-message result — nothing persists, no broker connection opens, no consumer starts. Every message resolves to exactly one terminal disposition:
  • landable — decoded, typed, and subscribed; the envelope carries the resolved type plus the verbatim decoded data.
  • duplicate — an at-least-once redelivery of a (topic, partition, offset) coordinate already seen in this batch; skipped so the message never ingests twice.
  • filtered — well-formed but the resolved type is outside your filter; skipped deliberately.
  • dead_letter — undecodable (invalid_json), not a JSON object (not_an_object), or the type path resolved to nothing (missing_type). Routed aside with structured errors so a poison message never blocks the partition.
Alongside the per-message dispositions, the result reports commit offsets — the highest safe-to-commit offset per partition. Every message at or below that offset reached a terminal disposition, so advancing the offset past it neither drops nor re-skips a record. That is how the engine maps onto Kafka’s at-least-once semantics: the (topic, partition, offset) coordinate doubles as a deterministic ingest id, stable across redeliveries, so the batch dedupes on it and a later ingest collapses on it. persisted: false in the response is deliberate — a dry run never writes.

​
4. Failure modes

The model is built so the known failure modes are each loud and bounded:
  • Poison messages — dead-letter with a structured error (invalid_json, not_an_object, missing_type) and the partition offset still advances past them. One bad record never stalls the partition.
  • Wrong type path — every message dead-letters with missing_type, so a misconfigured type_path surfaces on the first dry run, not in production.
  • Broker redelivery — the deterministic ingest id collapses duplicates within a batch and across polls.
  • Credential handling — the SASL password is write-only and encrypted at rest; nothing — not a read endpoint, not the dashboard, not an export — returns it.
  • Transport not yet consuming — the long-running broker consumer is a tracked follow-up gated behind an operator flag that defaults off. Saving a config with enabled: true prepares the subscription; it cannot start consumption. The dry run above is honest about that boundary: it validates the deterministic core the transport will call per poll, without pretending the transport runs.

​
5. Where it composes

Three sibling surfaces bookend the inbound path:
  • Export families model — the egress decision framework. A source is the reverse direction of the event-sink lane in that family: sinks stream out, sources consume in, and the two share one filter and envelope model.
  • Inbound webhook debug log — the per-request forensics plane for provider webhooks. Where webhook debugging answers “did the provider’s POST arrive and verify”, the source model is the durable, broker-based inbound path for events you originate yourself — the two inbound surfaces complement rather than overlap.
  • Event Sinks API — the outbound sibling’s field reference and the outbound mirror of the config model this page framed.

​
6. See also

Event sources inbound guide

The step-by-step for configuring brokers, topic, consumer group, and credentials, plus a consume-batch dry-run walkthrough.

Webhook fan-out and event sinks

The outbound model a source mirrors — one event taxonomy fanning out to buffer, webhooks, and sinks.

Export families model

Where the event-sink lane sits among ad-hoc CSV, vCon, reverse-ETL, and WORM egress.

Inbound webhook debug log

The pre-verify forensics plane for provider-side inbound traffic.