Skip to main content

​
CDP ad-audience incrementality

Activating a CDP segment to Meta, Google, TikTok, or another ad-network audience is easy to do; proving the spend worked is the hard part. This concept covers Orbit’s measurement primitive for activated audiences — a control holdout cohort plus a conversion-lift report — the layer that answers “did this activation cause incremental conversions, or would those members have converted anyway?” The ads activation guide sets up the route; this page explains the experiment contract it runs. Endpoint schemas live in the Ads API reference.

​
1. The incrementality question

Downstream revenue reads (ROAS on the attribution model) tell you what correlated with the ad activation — they cannot tell you what the spend caused. A segment of high-intent customers converts anyway; crediting the ad with their conversions is a selection artifact, not an effect. The measurement-serious answer — Braze’s Global Control Group, Meta’s Conversion Lift, Google’s conversion lift, every DSP’s lift study — is to compare the exposed audience against an unexposed baseline drawn from the same cohort. The difference between the two arms’ conversion rates is the incremental effect. Orbit carves that baseline directly into the activation route, per destination, as a tenant-owned setting.

​
2. Holdout mechanics

Each destination carries holdout_pct, a control percent in [0, 50], default 0 (off). Set it on the destination config:
On every activation of a destination with holdout_pct > 0, each submitted member is deterministically assigned a bucket from sha256("ad-audience-control:<audienceId>:<memberKey>") — where the member key is the member’s strongest stable identifier (normalized email > work email > phone > mobile ad id). Members whose bucket falls inside the percent are held out (control) and are NEVER uploaded to the ad network; the rest go to the treatment arm and upload normally. Rules that keep the holdout honest:
  • Deterministic and stable. The same person lands in the same arm on every activation of this destination, and is never accidentally exposed by a later sync. The hash is seeded per destination audience id, so a contact’s control status on one network cannot bias another.
  • Unidentifiable members route to treatment. A member whose identifiers all fail normalization cannot be assigned a stable bucket, so it uploads with treatment rather than silently dropping out of the population — held-out data loss is the noise incrementality must not manufacture.
  • The 50% ceiling. Values above 50 are clamped so a malformed config can never invert which arm is the experiment.
The run stores a counts-only breadcrumb — holdout percent, treatment count, control count — never the members, so the activation history stays bounded and carries no PII. Setting holdout_pct back to 0 disables the holdout from the next activation on; it does not retro-expose previously held-out members.

​
3. The lift report

After the observation window, post each arm’s measured conversions — joined from your analytics or warehouse — to the lift endpoint. The arm sizes come from the run’s stored breadcrumb, so the report is the causal control-vs-exposed comparison, not a self-reported guess:
The report is a Wald two-proportion test — the same statistics the campaigns pillar surfaces on its holdout card, so the two measurement surfaces report identically: A run activated with no holdout answers 409 — lift cannot be measured without a control arm. The latest report is persisted onto the activation run and returned from GET /api/v1/cdp/audience/activations/:id, so the measurement travels with the activation history.

​
4. Scope the segment so the holdout is meaningful

A lift report is only as strong as the experiment behind it — and sizing here is the same discipline the campaign-lifecycle holdout rules use:
  • Randomization needs a sizable arm. A control of, say, a few dozen members has variance too large for a significant read at any practical scale — the report will answer counts_only or is_significant: null until enough conversions accrue. Sizable enough is the difference between a 5% control on a 50,000-member segment (a readable 2,500-member control) and a 5% control on a 300-member segment (statistical noise dressed as an experiment).
  • The holdout protects against exposure, not eligibility. Members held out are skipped by this destination’s uploads; nothing prevents another destination, another network, or a live campaign from reaching them — treat the control as an unexposed-from-this-activation baseline and prefer a holdout spanning all paid-media routes to the segment when cross-channel contamination matters.
  • One percent per destination. A 5% holdout on Meta and a 5% holdout on Google carve independent control arms — the two are not synced. Keep the controlled population explicit per route, and recompute at the seeded-audience level to keep the assignments independent.

​
5. Integration with ad-audience activation

The holdout and lift pieces are built into the activation route itself — no separate experiment workspace to maintain:
  • Configure once. holdout_pct lives on the destination config next to the segment mapping and pre-activation gates (see the ads activation guide), so once set every activation of that destination carves the control automatically.
  • Attach on dispatch. POST /api/v1/cdp/audience/activate/:platform resolves the submitted members and partitions them — treatment uploads, control is skipped, and the breadcrumb is stored on the run.
  • Measure when ready. Fetch the run from GET /api/v1/cdp/audience/activations (listed without the per-member error feedback; GET /api/v1/cdp/audience/activations/:id carries the latest lift report), join the two arms’ conversions from your analytics, and post them to the lift endpoint. The report persists onto the run.
Because assignments hash a stable identifier, disabling and re-enabling the holdout does not re-randomize a member — a contact held out on the first activation stays held out across the destination’s re-syncs for as long as holdout_pct > 0.

​
See also