Skip to main content

​
A/B test a campaign

An A/B test splits a campaign’s audience across up to four message variants and reports delivery, open, click, and reply rates for each, so you can see which message performs better instead of guessing. Turn on A/B testing when you create the campaign, choose the variant split that fits the question you’re asking, and either let Orbit pick the winner or pick it yourself. This guide covers the variant model, the split options, how winners are chosen, and how to read results. For the full request and response schemas, see the campaigns API reference.
Experimentation sits beside governance, not over it. Every experiment assignment — a variant, a weighted split, or a hold-out control — still passes through the org-wide governance gates: per-contact frequency caps and the quiet-hours window run on the final recipient set, so a control cohort that receives no message also consumes no cap slot, and a winner promotion only decides which message sends, never whether it should. Size experiment cohorts from the governance-eligible audience (suppressed, capped, and opted-out contacts netted out), not the raw match count.

​
Prerequisites

  • A draft campaign with a message_template set. Variants are alternatives to that base message, so an A/B test needs one.
  • The campaign is still in draft or scheduled state. Variants are locked in before launch; changing the set mid-send invalidates the comparison.
If you have not written the base message yet, start in the campaign composer. A/B testing applies to every sending channel — SMS, WhatsApp, email, RCS, push.

​
How variants and splits work

Each variant carries a name, a message body, and a position in the set (variant_index, 0–3). A campaign accepts between two and four variants. Every recipient in the test is assigned to exactly one variant, so a recipient never receives two versions of the message. Traffic splitting has three modes:
  1. Even split. The default. When no variant carries a sample_percent, recipients distribute evenly across the set.
  2. Weighted split. Set sample_percent on every variant (integer percentages that sum to exactly 100) to give one variant more of the exploration traffic — a 70/30 or 50/30/20 split. The allocation is all-or-nothing: the API rejects a payload where only some variants carry a share, or where the shares do not sum to 100.
  3. Bandit (Thompson). Enable ab_bandit and mid-send traffic shifts toward whichever variant is currently leading, instead of holding a fixed split until a winner is declared. Use this on large sends where waiting for a verdict wastes opportunity; on small sends a fixed split usually reads more cleanly.
The set you send replaces the previous set in full — updating variants is a wholesale swap, not a merge.

​
Choose how a winner is picked

Two settings control the verdict:
  • Winner metric (ab_winner_metric): delivered, opened, clicked, or replied. Pick the metric that matches the campaign’s goal. A click-through campaign judged on replies picks the wrong winner; the default is the delivery rate, which measures deliverability rather than engagement.
  • Holdout (ab_holdout_percent, 1–50): a slice of the audience that receives no variant during the exploration phase, as a control group. Comparing the winner against the holdout afterwards tells you whether the message beat doing nothing, not just which variant beat the other.
Once the test evaluates, the winner is locked and subsequent sends use that variant. You can also decide manually at any point with Select the winning A/B variant — the chosen variant must belong to the campaign’s current variant set.

​
Auto-winner governance

Three optional knobs on the same variants call decide whether the winner-locking scheduler selects the winner automatically, and how strict the verdict must be:
  • ab_auto_promote (default true): when true, the winner-locking scheduler promotes the leading variant as soon as the thresholds below are met. Set false to lock every promotion behind the manual select-winner call — for example while a compliance review runs the test plan.
  • ab_min_sample_size (integer, 10–1,000,000; default 30): the fewest deliveries the leading variant must have before any verdict is considered. Raise it on high-traffic campaigns where an early lead on thin samples would otherwise lock prematurely.
  • ab_confidence_threshold (0.500–0.999; default 0.950): the Bayesian posterior confidence the leading variant must reach. Raise it toward 0.99 for a stricter verdict, lower it toward 0.90 when you’d rather lock sooner.
Each field is independently optional on the POST, and explicitly nullable — send null for one to reset it to its default. The results endpoint echoes the stored values plus a live auto_winner_progress snapshot (the gate values the evaluator currently applies, the current leader, and whether it would already lock), so a dashboard can render real progress toward an automatic verdict.

​
Generate variants with AI

Generate AI A/B message variants rewrites the campaign’s base message into two to four alternatives that test distinct angles — a different hook, value framing, or call to action — and returns them with a one-line rationale for each. Generation is suggestion-only: nothing commits to the campaign until you review and post the set you want. The generator stays grounded in your own copy — it rephrases claims the base message already makes rather than inventing offers, discounts, or urgency the original never carried — and keeps the identical personalization tokens and a single opt-out instruction, so a suggested variant is carrier-ready when the base message was.

​
Read the results

Get a campaign’s A/B results returns per-variant delivery, open, click, and reply rates, the total assigned recipients, the selected winner and when it was picked, the holdout percentage, and whether the bandit is on. A campaign with no A/B test returns an empty, disabled result set rather than an error, so a results panel renders cleanly either way. Rates divide by the metric’s denominator: delivery rate against sends, open, click, and reply rates against deliveries. A variant with zero sends reports zeros, not nulls.

​
Example: set a weighted three-way split

cURL
The response echoes the variant set the campaign now carries. From there, launch the campaign and watch Get a campaign’s A/B results as assignments accumulate. auto_winner_progress in that response tells you whether the leader has cleared the sample and confidence gates yet.

​
Troubleshooting

  • 422 on the variants call. Check that every body field satisfies the contract: two to four variants, each with a unique variant_index in 0–3; a sample_percent payload set on every variant and summing to exactly 100; ab_holdout_percent between 1 and 50; at least one field present. The error message names the failing rule.
  • Results look empty right after launch. Assignment rows appear as the send progresses. Calling the results endpoint before recipients are assigned returns the empty, disabled shape.
  • Winner selection rejects your variant id. The id must come from the campaign’s current variant set — a full-replace variants call invalidates ids from the previous set. Fetch the results or variants list and pick from there.
  • A/B editor does not appear on a cloned campaign. Cloning preserves the A/B toggle and variant set; if the original campaign had testing disabled, enable it on the clone first.

​
See also

  • Frequency caps — the per-contact rolling-window governance every experiment assignment still passes through.
  • Quiet hours — the org-wide send-window gate that narrows every send, experiment or not.