Skip to main content

​
Agent prompt A/B experiments: lift, holdout, and attribution

An experiment splits your agent’s live conversations between two saved prompt versions — a control (variant A) and a candidate (variant B) — and counts one conversion metric on each side. Because the contact-to-variant assignment is deterministic and sticky, every conversation lands in the same arm, which is what makes the lift attribution and significance numbers safe to act on. This guide walks the Experiment console end to end: start an experiment, read the live cards, decide when to stop, and promote the winner as a new version. Endpoint paths below are relative. Send them against https://api.orbit.devotel.io/api/v1. The experiment routes need agents:read / agents:write scope and the owner, admin, or developer role.

​
1. Where the console lives

Open Agents → your agent → Experiments. The tab carries four surfaces:
  • an active experiment card at the top (variant A vs variant B with per-arm conversion rates, the delta, and the significance state),
  • a Lift attribution panel (control vs treatment rates with 95% confidence intervals),
  • a Results panel (significance banner, cost per conversion, and the auto-eval quality comparison),
  • and a Batch simulation panel below, for pre-deploy gate runs.
A table of past experiments sits underneath with each run’s winner and stats. Anything you do in the console maps one-to-one onto the experiment routes, so the same loop works from the API — the examples below use both. Only owners, admins, and developers can see the Experiments tab. At most one experiment per agent can be active at a time; start a second one and the API rejects it with 409 EXPERIMENT_ALREADY_ACTIVE.

​
2. Start an experiment

You need two saved versions to split between. Freeze your current production prompt as a version — that is variant A, the holdout control — and save the candidate prompt as a second version. If neither exists yet, save them from the Versions tab (see Agent versions). Click Start experiment in the Experiments tab, or call POST /agents/{agentId}/experiments:
Set four things, and set them deliberately:
  • Variant B — the candidate version. traffic_split_pct is the percentage of new conversations routed to variant B (1–99; default 50). Everything else lands on variant A. Variants A and B must be different versions, otherwise the run is rejected with a 422.
  • Conversion metric — the one outcome that decides the winner:
    • reply_received converts on the contact’s first reply after the agent responds (an engagement read),
    • goal_completion converts when the agent fires a goal_fired tool call (a task read),
    • human_handoff_avoided converts when the conversation ends without an escalation (a containment read). Pick the metric that matches the change you are testing. Shortening the prompt? Measure containment. Reworking tool instructions? Measure goal completion.
  • The holdout — variant A is the frozen production prompt. During warmup (either arm below the assignment floor) every assignment uses your configured split; after warmup the splitter adapts toward the leading arm but always keeps both arms exploring, and already-assigned contacts never re-roll.
The response is 201 with the experiment row including id, traffic_split_pct, and started_at. Keep the id — every readback route hangs off it.

​
3. Live readback

The top-of-page active experiment card refreshes the per-arm numbers as conversations accumulate. The same numbers are on GET /agents/{agentId}/experiments/{experimentId}:

​
Lift attribution

The Lift attribution panel (and GET /agents/{agentId}/ab/{experimentId}/lift) treats variant A as the control: variant A’s lift is pinned at zero, and variant B carries the relative lift versus control with a p-value and a 95% Wald confidence interval on each arm. Read this panel daily — it answers “how much better (or worse) is the candidate right now,” which a bare rate pair cannot.

​
Batch simulation

The Batch simulation panel (and POST /agents/{agentId}/batch-simulation) replays a suite of scripted multi-turn journeys against a candidate version in sandbox mode and returns an aggregate gate report: pass rate, p50/p95/max latency, and a cost projection. Run it before you start an experiment, or before promoting, to catch a candidate that passes live traffic but breaks a scripted edge case. A concrete example: you compare two refund-prompt versions on the same dataset. Variant A handles contacts at a 61% containment rate, variant B at 68%. The lift panel reports +11.5% relative lift on B with p = 0.03 and 95% CI [1.2, 12.4] points, so the winner panel highlights variant B. Before promoting, run the batch suite — if your “chargeback edge case” journey fails on variant B’s tool assertion, the experiment’s raw lift was not the whole story.

​
Auto-checks before you promote

The detail route surfaces a promotion_recommendation, and GET /agents/{agentId}/experiments/{experimentId}/auto-promotion runs the full gate: winner-by-posterior plus the cost guardrail. Promotion is only ever operator-pulled — neither the console nor any background job flips the winner automatically.

​
4. Stop criteria

End an experiment when the numbers stop moving and one of these holds:
  • Variant B clears significance and the gate. Promote B (step 6).
  • Variant A holds, or B is a wash. End the experiment, keep A live, and iterate on the candidate before another run.
  • The gate objects. If B leads on conversion but costs materially more per conversion, or fails the batch simulation, end without promoting.
End it with End experiment in the console or POST /agents/{agentId}/experiments/{experimentId}/end — optionally stamping the winner in the same call. Ending never touches the live agent row. For the holdout discipline itself — freezing the baseline version, scoring both arms against the same dataset, and re-baselining after any dataset edit — see Agent evals: datasets, runs, and pass-rate gates, section “Holdout vs experiment.”

​
5. Significance vs novelty

The Results panel (and GET /agents/{agentId}/experiments/{experimentId}/results) is the statistical-readout surface. It reports, per arm: conversion rate with the 95% confidence interval, the delta in percentage points, a two-proportion significance test (significant_at_95, significant_at_99), a minimum-sample progress bar, plus cost-per-conversion for both arms and the auto-eval quality comparison. Two failure modes to guard against:
  • Calling it early. A big lift on day one is usually novelty plus a small denominator. Wait until the panel reports the minimum sample met and significant_at_95 holds — the promote button stays gated until then for exactly this reason.
  • Calling it on conversion only. A variant that wins on conversions while doubling cost per conversion, or dropping the auto-eval quality score, is not a winner. Read all three rows on the Results panel before the decision.

​
6. Promote the winner as a version

Promotion copies the winning version’s full snapshot — prompt, model, tools, knowledge bases, safety config — onto the live agent row, bumps the agent’s version counter, and mints a new history row so you can roll back to the experiment state. Nothing auto-promotes; you always pull the trigger. From the console, click Promote winning variant on the active-experiment card or either panel. From the API:
The response carries new_version — the freshly minted version row — and the experiment is ended and stamped with the winner if it was still running. If the winning version was deleted in the meantime, the call returns 409 WINNER_VERSION_GONE and the live agent is untouched. Promotion writes the whole snapshot in one go, which is correct for a winner you already validated — but it is still a full-traffic change. If you want a canary ladder instead (10% → 50% → 100% with automatic rollback), route the winner through the rollout pipeline rather than promoting directly.

​
7. Failure modes

  • A start call returns 409 EXPERIMENT_ALREADY_ACTIVE. One experiment per agent at a time — end the running one first.
  • A start call returns 422 on the versions. Either variant id does not belong to this agent, or both variants reference the same version. Save two distinct versions and retry.
  • The lift panel shows vanity movement. Before the minimum sample is met, lift swings with each conversation. Check the Results panel’s progress bar, not the lift bar.
  • Promote returns 409 WINNER_VERSION_GONE. Someone deleted the winning version after the experiment ended. Pick the survivor, or end the experiment and re-save the winner from version history.
  • No assignments accumulate. The agent is not receiving live traffic. Experiments only observe real conversations — use the batch simulation panel if you need a readout before traffic exists.

​
See also