> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent prompt A/B experiments: lift, holdout, and attribution

> Split live agent traffic between two prompt versions against a holdout control, read daily lift attribution and significance, and promote the winner as a new version.

# Agent prompt A/B experiments: lift, holdout, and attribution

An experiment splits your agent's live conversations between two saved prompt
versions — a control (variant A) and a candidate (variant B) — and counts one
conversion metric on each side. Because the contact-to-variant assignment is
deterministic and sticky, every conversation lands in the same arm, which is
what makes the lift attribution and significance numbers safe to act on. This
guide walks the Experiment console end to end: start an experiment, read the
live cards, decide when to stop, and promote the winner as a new version.

Endpoint paths below are relative. Send them against
`https://api.orbit.devotel.io/api/v1`. The experiment routes need
`agents:read` / `agents:write` scope and the owner, admin, or developer role.

## 1. Where the console lives

Open **Agents → your agent → Experiments**. The tab carries four surfaces:

* an **active experiment card** at the top (variant A vs variant B with
  per-arm conversion rates, the delta, and the significance state),
* a **Lift attribution** panel (control vs treatment rates with 95%
  confidence intervals),
* a **Results** panel (significance banner, cost per conversion, and the
  auto-eval quality comparison),
* and a **Batch simulation** panel below, for pre-deploy gate runs.

A table of past experiments sits underneath with each run's winner and stats.
Anything you do in the console maps one-to-one onto the experiment routes, so
the same loop works from the API — the examples below use both.

Only owners, admins, and developers can see the Experiments tab. At most one
experiment per agent can be active at a time; start a second one and the API
rejects it with `409 EXPERIMENT_ALREADY_ACTIVE`.

## 2. Start an experiment

You need two saved versions to split between. Freeze your current production
prompt as a version — that is variant A, the holdout control — and save the
candidate prompt as a second version. If neither exists yet, save them from
the **Versions** tab (see [Agent versions](/agents/agent-versions)).

Click **Start experiment** in the Experiments tab, or call
`POST /agents/{agentId}/experiments`:

```bash theme={null}
curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/experiments \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "refund-flow shorter prompt",
    "variant_a_version_id": "apv_control_7",
    "variant_b_version_id": "apv_candidate_12",
    "traffic_split_pct": 50,
    "conversion_metric": "human_handoff_avoided"
  }'
```

Set four things, and set them deliberately:

* **Variant B** — the candidate version. `traffic_split_pct` is the
  percentage of new conversations routed to variant B (1–99; default 50).
  Everything else lands on variant A. Variants A and B must be different
  versions, otherwise the run is rejected with a `422`.
* **Conversion metric** — the one outcome that decides the winner:
  * `reply_received` converts on the contact's first reply after the agent
    responds (an engagement read),
  * `goal_completion` converts when the agent fires a `goal_fired` tool call
    (a task read),
  * `human_handoff_avoided` converts when the conversation ends without an
    escalation (a containment read).
    Pick the metric that matches the change you are testing. Shortening the
    prompt? Measure containment. Reworking tool instructions? Measure goal
    completion.
* **The holdout** — variant A is the frozen production prompt. During warmup
  (either arm below the assignment floor) every assignment uses your
  configured split; after warmup the splitter adapts toward the leading arm
  but always keeps both arms exploring, and already-assigned contacts never
  re-roll.

The response is `201` with the experiment row including `id`,
`traffic_split_pct`, and `started_at`. Keep the id — every readback route
hangs off it.

## 3. Live readback

The top-of-page **active experiment** card refreshes the per-arm numbers as
conversations accumulate. The same numbers are on
`GET /agents/{agentId}/experiments/{experimentId}`:

```bash theme={null}
curl https://api.orbit.devotel.io/api/v1/agents/agent_abc123/experiments/agentExperiment_a1b2c3 \
  -H "X-API-Key: dv_live_sk_..."
```

### Lift attribution

The **Lift attribution** panel (and
`GET /agents/{agentId}/ab/{experimentId}/lift`) treats variant A as the
control: variant A's lift is pinned at zero, and variant B carries the
relative lift versus control with a p-value and a 95% Wald confidence
interval on each arm. Read this panel daily — it answers "how much better
(or worse) is the candidate right now," which a bare rate pair cannot.

### Batch simulation

The **Batch simulation** panel (and
`POST /agents/{agentId}/batch-simulation`) replays a suite of scripted
multi-turn journeys against a candidate version in sandbox mode and returns
an aggregate gate report: pass rate, p50/p95/max latency, and a cost
projection. Run it before you start an experiment, or before promoting, to
catch a candidate that passes live traffic but breaks a scripted edge case.

A concrete example: you compare two refund-prompt versions on the same
dataset. Variant A handles contacts at a 61% containment rate, variant B at
68%. The lift panel reports `+11.5%` relative lift on B with
`p = 0.03` and 95% CI `[1.2, 12.4]` points, so the winner panel highlights
variant B. Before promoting, run the batch suite — if your "chargeback
edge case" journey fails on variant B's tool assertion, the experiment's
raw lift was not the whole story.

### Auto-checks before you promote

The detail route surfaces a `promotion_recommendation`, and
`GET /agents/{agentId}/experiments/{experimentId}/auto-promotion` runs the
full gate: winner-by-posterior plus the cost guardrail. Promotion is only
ever operator-pulled — neither the console nor any background job flips the
winner automatically.

## 4. Stop criteria

End an experiment when the numbers stop moving and one of these holds:

* **Variant B clears significance and the gate.** Promote B (step 6).
* **Variant A holds, or B is a wash.** End the experiment, keep A live, and
  iterate on the candidate before another run.
* **The gate objects.** If B leads on conversion but costs materially more
  per conversion, or fails the batch simulation, end without promoting.

End it with **End experiment** in the console or
`POST /agents/{agentId}/experiments/{experimentId}/end` — optionally stamping
the winner in the same call. Ending never touches the live agent row.

For the holdout discipline itself — freezing the baseline version, scoring
both arms against the same dataset, and re-baselining after any dataset edit
— see
[Agent evals: datasets, runs, and pass-rate gates](/guides/agent-evals-datasets-loop),
section "Holdout vs experiment."

## 5. Significance vs novelty

The **Results** panel (and
`GET /agents/{agentId}/experiments/{experimentId}/results`) is the
statistical-readout surface. It reports, per arm: conversion rate with the
95% confidence interval, the delta in percentage points, a two-proportion
significance test (`significant_at_95`, `significant_at_99`), a minimum-sample
progress bar, plus cost-per-conversion for both arms and the auto-eval
quality comparison.

Two failure modes to guard against:

* **Calling it early.** A big lift on day one is usually novelty plus a small
  denominator. Wait until the panel reports the minimum sample met and
  `significant_at_95` holds — the promote button stays gated until then for
  exactly this reason.
* **Calling it on conversion only.** A variant that wins on conversions while
  doubling cost per conversion, or dropping the auto-eval quality score, is
  not a winner. Read all three rows on the Results panel before the decision.

## 6. Promote the winner as a version

Promotion copies the winning version's full snapshot — prompt, model, tools,
knowledge bases, safety config — onto the live agent row, bumps the agent's
version counter, and mints a new history row so you can roll back to the
experiment state. Nothing auto-promotes; you always pull the trigger.

From the console, click **Promote winning variant** on the active-experiment
card or either panel. From the API:

```bash theme={null}
curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/experiments/agentExperiment_a1b2c3/promote-winner \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{ "winner": "b", "note": "68% containment at 95% significance, cost-neutral" }'
```

The response carries `new_version` — the freshly minted version row — and the
experiment is ended and stamped with the winner if it was still running. If
the winning version was deleted in the meantime, the call returns
`409 WINNER_VERSION_GONE` and the live agent is untouched.

Promotion writes the whole snapshot in one go, which is correct for a winner
you already validated — but it is still a full-traffic change. If you want a
canary ladder instead (10% → 50% → 100% with automatic rollback), route the
winner through the
[rollout pipeline](/guides/ai-agent-rollout-pipeline) rather than
promoting directly.

## 7. Failure modes

* **A start call returns `409 EXPERIMENT_ALREADY_ACTIVE`.** One experiment
  per agent at a time — end the running one first.
* **A start call returns `422` on the versions.** Either variant id does not
  belong to this agent, or both variants reference the same version. Save two
  distinct versions and retry.
* **The lift panel shows vanity movement.** Before the minimum sample is met,
  lift swings with each conversation. Check the Results panel's progress bar,
  not the lift bar.
* **Promote returns `409 WINNER_VERSION_GONE`.** Someone deleted the winning
  version after the experiment ended. Pick the survivor, or end the
  experiment and re-save the winner from version history.
* **No assignments accumulate.** The agent is not receiving live traffic.
  Experiments only observe real conversations — use the batch simulation
  panel if you need a readout before traffic exists.

## See also

* [Agent versions](/agents/agent-versions) — freeze the control and candidate
  you split between.
* [Agent evals: datasets, runs, and pass-rate gates](/guides/agent-evals-datasets-loop)
  — the holdout concept and offline scoring loop that pairs with experiments.
* [Safely Roll Out an AI Agent](/guides/ai-agent-rollout-pipeline) — the
  canary ladder for a staged winner promotion.
* [Agent containment dashboard](/guides/agent-containment-dashboard) — read
  containment trends after a promotion lands.
* [Agent evaluation endpoints](/api-reference/endpoints/agents) — the full
  route reference.