> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# CDP streaming destinations: Kafka and Kinesis real-time egress

> When to stream Orbit events to a Kafka topic or Kinesis stream instead of a webhook, how the normalized stream record is shaped, how the partition key is derived for per-user ordering, and which layer owns the delivery.

# CDP streaming destinations: Kafka and Kinesis egress

The destination kinds covered in the [destinations guide](/guides/cdp-destinations)
are **webhook receivers** — Orbit POSTs each event to an HTTPS endpoint you own
and retries on failure. A **streaming destination** is a different egress
shape: every event is produced onto a message bus you operate — a **Kafka
topic** or an **AWS Kinesis stream** — so real-time consumers (Flink jobs,
service streamers, Spark Structured Streaming, Kinesis consumers) can
subscribe to Orbit's event flow continuously, and replay it by offset or by
shard position instead of by webhook payload.

This page defines the record model for streaming destinations: when to pick
streaming over webhook egress, what one record looks like on the bus, how the
partition or shard key is derived, and which layer of the stack owns the
actual producer connection vs. the record formatting.

## Section 1 — Streaming vs. webhook egress

Webhook egress and streaming egress solve for different consumers, and the
trade-off is worth naming before you configure either:

| Property          | Webhook egress                                  | Streaming egress                                                  |
| ----------------- | ----------------------------------------------- | ----------------------------------------------------------------- |
| Receiver          | Any HTTPS endpoint Orbit can POST to            | A Kafka topic or Kinesis stream you own                           |
| Consumer coupling | Receiver must be reachable from Orbit           | Consumers poll or subscribe to the bus                            |
| Retry semantics   | Orbit retries failed payloads (circuit breaker) | Once produced, the bus holds the record — consumers ack by offset |
| Replay            | Re-drive failed payloads from the delivery log  | Consumers seek back to an offset / shard position                 |
| Ordering          | None — each event is an independent POST        | Per-user ordering inside a partition / shard                      |

Pick **webhook** when your downstream is a receiver (an HMAC-verifying HTTPS
endpoint, a data-warehouse webhook, a CDP vendor intake). The circuit-breaker
and DLQ semantics of that path are covered in the
[destinations guide](/guides/cdp-destinations).

Pick **streaming** when your downstream is a bus — Kafka connect pipelines,
stream-processing topologies, or Kinesis consumers — and you want the event
stream delivered into the broker's storage rather than aimed at a live
endpoint. Streaming destinations are configured on the
**Integrations → CDP → Streaming** tab; the API surface behind it is
`GET`/`PATCH /api/v1/cdp/streaming-destinations/:provider` (provider `kafka`
or `kinesis`, owner/admin/developer roles, subject to the same role gating
the webhook destinations use).

Two capabilities are deliberately scoped off the webhook path: **ordering**
(a webhook POST carries one event; per-user ordering needs a keyed partition)
and **bus semantics** (seek, replay, consumer groups).

## Section 2 — The normalized stream record

Every event a streaming destination emits is normalized into one shape before
it is adapted to the provider. The normalization is a **pure formatter** — no
broker imports, no network calls, no SDK dependencies — so the record you
configure against is testable without standing up Kafka:

| Field          | Holds                                                                                                                  |
| -------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `provider`     | `kafka` or `kinesis`                                                                                                   |
| `target`       | The topic (Kafka) or stream (Kinesis) name                                                                             |
| `partitionKey` | The partition / shard routing key (see Section 3)                                                                      |
| `value`        | The full CDP event envelope, serialized to UTF-8 JSON                                                                  |
| `headers`      | Config string→string Kafka headers, plus a derived `content-type` and the event `type` for cheap consumer-side routing |

The `value` is the **same JSON bytes the webhook path would sign** — the full
canonical CDP envelope (`type`, `event`, `userId`/`anonymousId`, `properties`,
`context`). A consumer sharing one parser across webhook and streaming
delivery is the design intent: you can re-point a webhook receiver onto the
bus without re-writing the parse side, and vice versa.

The config you pass to the PATCH endpoint is validated by the same schema the
server applies, so a malformed topic or stream name, an out-of-range header
map, or an invalid `partition_key_path` is rejected with a 400 before a
record reaches the producer:

```json theme={null}
{
  "destination": {
    "provider": "kafka",
    "topic": "orbit-events",
    "partition_key_path": "userId",
    "headers": {
      "envelope-version": "1"
    }
  }
}
```

Kafka topic constraints (charset `[A-Za-z0-9._-]`, 1–249 chars, never the
bare `.` or `..`) and Kinesis stream-name constraints (`[A-Za-z0-9_.-]`,
1–128 chars) are enforced on both sides of the config boundary — same schema
as the formatter, so a rejected shape can never survive on disk.

## Section 3 — Partition and shard key derivation

Downstream consumers depend on **per-user ordering** — every event for one
customer lands on the same Kafka partition / Kinesis shard so a consumer
group processes them in sequence. That guarantee rides on a **deterministic
partition key**, derived by the same code path your config and the producer
agree on:

1. **Explicit `partition_key_path`** — if you set one in the config and it
   resolves to a scalar `string` or `number` in the event, that value wins.
2. **Identity fallback** — otherwise the shared fallback set:
   `userId`/`user_id` → `anonymousId`/`anonymous_id` →
   `contactId`/`contact_id` → `messageId`/`message_id`, resolved in order,
   first non-empty match. This covers every event Orbit ingests, so ordering
   is user-scoped by default.
3. **Stable hash fallback** — otherwise the SHA-256 hex of the serialized
   envelope. This gives a fully-anonymous event a stable home without ever
   emitting a null key.

Two edge behaviors are fixed intentionally:

* **Length cap.** Kinesis caps `PartitionKey` at 256 chars, and Kafka has a
  similar boundary. An overlong key isn't silently truncated — that would
  smear a user across partitions. It's replaced by the full SHA-256 hex of
  the raw value (64 chars), so a long explicit value still routes
  deterministically without a collision blind spot.
* **Dot-path safety.** The `partition_key_path` resolver looks up
  `user.id`, `properties.plan`, whatever you configure — it resolves only
  object keys, never `__proto__`-adjacent lookups, so a path into the event
  payload cannot climb into the prototype chain.

Key derivation is per destination, and the fallback cascade (identity →
hash) is identical across producers — two destinations never disagree over
which key one event carries.

## Section 4 — Provider adapters

The normalized record is adapted to the producer SDK expected by each
provider, at the boundary where the record config is already resolved:

* **Kafka** — the record is handed to a `kafkajs`-satisfying producer as
  `producer.send({ topic, messages })`, one message per record, with the
  string→string headers attached:

  ```js theme={null}
  {
    topic: "orbit-events",
    messages: [
      {
        key: "user_42f8",
        value: "{\"type\":\"track\",\"event\":\"Order Completed\",…}",
        headers: { "content-type": "application/json", "envelope-version": "1" }
      }
    ]
  }
  ```

* **Kinesis** — the record is handed to the AWS SDK as a `PutRecords` entry:

  ```js theme={null}
  {
    Data: "{\"type\":\"track\",\"event\":\"Order Completed\",…}",
    PartitionKey: "user_42f8"
  }
  ```

The adapter is intentionally thin: nothing provider-neutral is lost in the
map, and no provider-specific shape is leaked back into the normalized
record. Headers exist for Kafka only because Kinesis has no per-record header
concept — the Kinesis entry carries `Data` and `PartitionKey`, the two fields
AWS expects.

## Section 5 — Delivery ownership: the boundary between formatter and worker

Streaming destinations ship as two cooperating layers, and knowing where the
boundary sits keeps configuration separate from operation:

* **The formatter is a pure primitive.** Mapping an event + config to a
  normalized record, and adapting it to the provider shape, has no network
  effect and no broker dependency — that is why `partition_key_path`
  validation and key derivation happen deterministically in a unit-testable
  seam.
* **Connection, batching, and retry live with the producer layer.** The
  producer worker owns the sequence that matters for health: broker connect,
  batch aggregation, append-retries, and run-status bookkeeping. The
  `GET`/`PATCH` endpoints store **operator-owned config** (destination,
  credentials, enable flag) and surface **worker-owned run-status** fields
  (`last_run_at`, `last_run_status`, `last_produced_count`, `last_error`,
  `consecutive_failures`) as read-only — they are owned the same way both
  sides.

The upshot for planning: right now, configure through the
`GET`/`PATCH /api/v1/cdp/streaming-destinations/:provider` arm; the endpoint
and the dashboard **Streaming** tab hold config and worker-owned health in
one place, and the worker reads the same settings. Credentials (Kafka SASL
password / AWS secret access key) are encrypted at rest, and the GET endpoint
answers with a `credentials_configured` boolean — the secret never round
trips in plaintext, and the audit log records changed key names, never
values.

Three controls here are tenant-owned, matching Orbit's compliance posture —
no platform hard gate, only tenant toggles: the **role-gated PATCH** (owner /
admin / developer), the **encrypted credential storage**, and the
**operator-set enable flag**. Compliance is the tenant's to configure.

## Cross-references

* [Where events land](/concepts/cdp-event-model#section-6--where-events-land) —
  Section 6 of the event model now lists streaming destinations alongside
  webhooks, segments, computed traits, funnels, attribution, and DSAR.
* [CDP destinations guide](/guides/cdp-destinations) — the webhook side of
  the same console, with the circuit breaker and DLQ.
* [Source and destinations console](/guides/cdp-source-and-destinations) —
  the two halves of Integrations → CDP, inbound secret and outbound
  receivers.
* [Reverse-ETL destination model](/concepts/reverse-etl-destination-model) —
  scheduled replicated tables vs. real-time bus egress.
* [Export families model](/concepts/export-families-model) — the map of
  export families: which one answers "replication vs. artifact vs. stream."