​

CDP streaming destinations: Kafka and Kinesis egress

​

Section 1 — Streaming vs. webhook egress

​

Section 2 — The normalized stream record

{
  "destination": {
    "provider": "kafka",
    "topic": "orbit-events",
    "partition_key_path": "userId",
    "headers": {
      "envelope-version": "1"
    }
  }
}

​

Section 3 — Partition and shard key derivation

1. Explicit partition_key_path — if you set one in the config and it resolves to a scalar string or number in the event, that value wins.
2. Identity fallback — otherwise the shared fallback set: userId/user_id → anonymousId/anonymous_id → contactId/contact_id → messageId/message_id, resolved in order, first non-empty match. This covers every event Orbit ingests, so ordering is user-scoped by default.
3. Stable hash fallback — otherwise the SHA-256 hex of the serialized envelope. This gives a fully-anonymous event a stable home without ever emitting a null key.

• Length cap. Kinesis caps PartitionKey at 256 chars, and Kafka has a similar boundary. An overlong key isn’t silently truncated — that would smear a user across partitions. It’s replaced by the full SHA-256 hex of the raw value (64 chars), so a long explicit value still routes deterministically without a collision blind spot.
• Dot-path safety. The partition_key_path resolver looks up user.id, properties.plan, whatever you configure — it resolves only object keys, never __proto__-adjacent lookups, so a path into the event payload cannot climb into the prototype chain.

​

Section 4 — Provider adapters

• Kafka — the record is handed to a kafkajs-satisfying producer as producer.send({ topic, messages }), one message per record, with the string→string headers attached:
  {
    topic: "orbit-events",
    messages: [
      {
        key: "user_42f8",
        value: "{\"type\":\"track\",\"event\":\"Order Completed\",…}",
        headers: { "content-type": "application/json", "envelope-version": "1" }
      }
    ]
  }
  
• Kinesis — the record is handed to the AWS SDK as a PutRecords entry:
  {
    Data: "{\"type\":\"track\",\"event\":\"Order Completed\",…}",
    PartitionKey: "user_42f8"
  }
  

​

Section 5 — Delivery ownership: the boundary between formatter and worker

• The formatter is a pure primitive. Mapping an event + config to a normalized record, and adapting it to the provider shape, has no network effect and no broker dependency — that is why partition_key_path validation and key derivation happen deterministically in a unit-testable seam.
• Connection, batching, and retry live with the producer layer. The producer worker owns the sequence that matters for health: broker connect, batch aggregation, append-retries, and run-status bookkeeping. The GET/PATCH endpoints store operator-owned config (destination, credentials, enable flag) and surface worker-owned run-status fields (last_run_at, last_run_status, last_produced_count, last_error, consecutive_failures) as read-only — they are owned the same way both sides.

​

Cross-references

• Where events land — Section 6 of the event model now lists streaming destinations alongside webhooks, segments, computed traits, funnels, attribution, and DSAR.
• CDP destinations guide — the webhook side of the same console, with the circuit breaker and DLQ.
• Source and destinations console — the two halves of Integrations → CDP, inbound secret and outbound receivers.
• Reverse-ETL destination model — scheduled replicated tables vs. real-time bus egress.
• Export families model — the map of export families: which one answers “replication vs. artifact vs. stream.”