​

Holdouts and uplift measurement

1. “Did sending lift conversion?” — the campaign-wide holdout. You set a percentage of the eligible audience that receives nothing, then compare the senders’ conversion against that do-nothing control.
2. “Did optimizing WHEN we send lift conversion?” — the smart-send holdout, available on campaigns running recipient-optimal send-time optimization (smart_send). You hold a percentage of contacts out of the time optimization; they receive the campaign immediately in natural batch order while the rest receive their predicted best-time window. Comparing the two isolates the value of the scheduling optimization above the campaign’s baseline effect.

​

Enable a holdout

​

Campaign-wide holdout — did sending lift conversion?

curl -X PATCH "https://api.orbit.devotel.io/api/v1/campaigns/{id}" \
  -H "X-API-Key: $ORBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "variables": { "campaign_holdout_percent": 10 } }'

​

Smart-send holdout — did optimizing WHEN we send lift conversion?

curl -X PATCH "https://api.orbit.devotel.io/api/v1/campaigns/{id}" \
  -H "X-API-Key: $ORBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "variables": { "smart_send_holdout_pct": 10 } }'

​

How assignment stays random

​

Read lift: the two endpoints

• GET /campaigns/:id/holdout-lift — did sending lift conversion? The card reports has_holdout: false and tracking_method: "no_holdout" on an un-configured campaign; counts_only when the control cohort was carved out but per-arm conversions haven’t been joined yet; and a full lift + confidence interval once both arms’ conversions are known.
• GET /campaigns/:id/smart-send-holdout-lift — did optimizing when we send lift conversion? On a campaign not using recipient-optimal scheduling (or with no smart-send holdout set) the card reports configured: false and tracking_method: "no_holdout" — a benign read state, not an error.

​

Outcome metrics on the smart-send card

• delivered — the contact’s message reached a terminal delivered status.
• read — a WhatsApp/RCS read receipt, or the email open-pixel stamp (the same predicate the deliverability dashboard’s “Read Rate” card uses).
• replied — an inbound message arrived on the contact’s conversation attributed to this campaign.

curl "https://api.orbit.devotel.io/api/v1/campaigns/{id}/smart-send-holdout-lift" \
  -H "X-API-Key: $ORBIT_API_KEY"

import { Orbit } from "@devotel-orbit/node";

const orbit = new Orbit({ apiKey: process.env.ORBIT_API_KEY });

const result = await orbit.request(
  "GET",
  "/campaigns/{id}/smart-send-holdout-lift",
);

if (result.data.configured) {
  console.log(
    "Click lift:",
    result.data.lift.absolute_lift_pct,
    "pp (significant:",
    result.data.lift.is_significant,
    ")",
  );
}

​

A worked workflow

1. Set both flags before launch. Say, a 10% campaign-wide holdout plus a 10% smart-send holdout on a recipient-optimal campaign. Launch.
2. Read the campaign-wide lift. GET /campaigns/:id/holdout-lift — until its is_significant reads true, treat the campaign’s effect as unproven vs. doing nothing.
3. Read the optimization lift. GET /campaigns/:id/smart-send-holdout-lift — check lift (clicks) and each card under metrics (delivered/read/replied).
4. Decide whether the optimization earns its keep. When the lift’s confidence interval spans zero, the optimization is not demonstrably moving conversion at your current volume — either widen the audience, accept the un-optimized schedule, or re-run on a larger send. When the interval clears zero, keep the optimizer enabled and read the same card on future sends to confirm the effect holds.