Skip to main content

​
Batch-load CDP data from CSV, JSON, or JSONL files

POST /api/v1/cdp/file-ingest is the non-SDK bulk-load path into the CDP. You post a file’s raw text — CSV, a JSON array, or newline-delimited JSON — and each row lands through the same ingestion primitives the live tracking surface uses: the same identity rules, the same deterministic message-id dedupe, and the same GDPR right-to-erasure gate. Rows that name people land on profiles; rows that name events land on the event stream. This is the connect-or-validate surface a migration script, an onboarding runbook, or a backfill job calls. It is deliberately synchronous and bounded: one request carries up to 200 rows and returns per-row errors and counters, so you always know exactly which rows landed and which need fixing.

​
1. When to use file-ingest vs the signed SDK surface

The two ingest paths exist for different jobs. Pick deliberately. A few rules of thumb:
  • Migrations and onboarding. A migration script that holds the export in memory should call file-ingest once per chunk; it is far simpler than standing up a signer for a one-shot job.
  • Backfills. Events you replay from a warehouse dump keep their original timestamp per row, so the event stream records when they actually happened, not when you uploaded.
  • Live traffic. Never feed steady app traffic through file-ingest — the 200-row cap and the operator-role requirement make it the wrong tool. Use the signed surface.
For the background, the CDP loop is documented end to end in first CDP activation and the HMAC contract is spelled out in sign CDP ingest requests.

​
2. The endpoint

Authenticate with your workspace API key (Clerk session also works). The route requires the owner, admin, or developer role, and the rate limit is intentionally tight (10 calls per minute) — this is a bulk operator action, not a hot path.
The response returns counters and a per-row error list — see section 4.

​
Row mapping

The reader accepts both nested and flat shapes. Events. A row needs an event name in event (also accepted: event_name, eventName). userId / user_id and anonymousId / anonymous_id are optional — an event with neither still lands, it just does not link to a contact. Everything that is not a reserved key folds into properties, and a timestamp column sets the event time. Profiles. A row needs at least one identifier: userId, anonymousId, or an email / phone trait. Non-reserved keys fold into traits. Ship one of those identifiers on every row — otherwise the row has nothing to bind a contact to, and it comes back in errors. CSV with a header row is the most convenient shape: the first row names the keys, and the RFC-4180 reader handles quoted fields, embedded commas and newlines, and \r\n endings.

​
3. Dry-run semantics

dry_run: true runs the full pipeline — decode, column-count checks, domain mapping — and returns what would ingest, with zero writes and no audit entry. Use it to shake a file before you commit rows to the workspace:
  1. Post the file with dry_run: true.
  2. Compare valid_rows against your expected count and read every entry in errors.
  3. Fix the flagged rows, re-dry-run.
  4. Post with dry_run: false (or omit the flag).
Nothing about the validation differs between the two modes — a row that fails dry-run fails the real run, and a file that validates clean lands clean.

​
4. Bounds: 200 rows per call, 1 MiB per request

Two hard ceilings keep this endpoint synchronous and safe on the request path:
  • Row cap. At most 200 rows are read per request. When the file holds more, the response sets truncated: true and the remainder is skipped — chunk larger files by the caller and page through the chunks.
  • Byte cap. content is bounded by the inline ceiling — 1 MiB (1,048,576 bytes) — the same cap the live tracking surface enforces. A request over the cap is rejected at validation with a 400.
Splitting is safe to retry: events you replay carry their own message_id if the file has one, and when it does not, the workspace derives a deterministic message id from the row bytes — so re-uploading a chunk dedupes instead of double-writing. Response counters tell you which outcome every row reached:

​
5. GDPR right-to-erasure gate, per row

Every non-dry-run row passes the erasure gate before a write. If the row’s userId or anonymousId, or its email / phone trait, matches a contact your workspace has erased (right-to-erasure under GDPR Article 17), the row counts under tombstoned and nothing is written for it — a re-upload can never resurrect an erased profile. For profiles, matching an erased identifier declines the upsert entirely. For events, the row is skipped before resolution. Either way the erasure record is authoritative and the counter tells you exactly how many rows the gate declined. Erase and re-export deliberately: a migration file that still contains erased people gets them declined, not resurrected.

​
6. Where the parser goes next

The decode-and-map half of this endpoint is a pure, dependency-free parser: no object-storage SDK, no transport, no schema changes. The scheduled bucket reader for object-storage source drops reuses this same parser once that transport lands — so the CSV, JSON-array, and JSONL shapes you post today are exactly the shapes the scheduled reader accepts when it ships. Parquet stays out of scope for now: it is a binary format that needs a real dependency, and the text formats cover the common export shapes.

​
7. Troubleshooting

​
See also