> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Safely Roll Out an AI Agent

> Move a new prompt, model, or knowledge update into production with persona simulation, retrieval benchmarks, shadow comparison, guardrails, canary stages, regression checks, and a monitored rollback path.

# Safely Roll Out an AI Agent

Prompt and model changes are production changes. The same discipline you
apply to a code deploy — test offline, inspect against a stop list, roll a
slice of traffic, watch the quality numbers, keep a one-call rollback —
applies to an agent's `system_prompt`, `model`, or knowledge-base update.
Orbit already ships each stage; this guide wires them into one pipeline.

The endpoint paths below are relative. Send them against
`https://api.orbit.devotel.io/api/v1`.

## What the pipeline covers

* Choosing a model/runtime profile (chat vs voice vs retrieval).
* Grounding the agent in your knowledge sources, before launch.
* Defining the guardrail policy that shapes what can be said.
* Offline comparison of the candidate against the prompt in production.
* Graduated rollout that shifts one small traffic slice at a time.
* Automatic regression runs on every publish.
* Live scoring, drift detection, and a human-in-the-loop when you want it.
* A clean fallback to a person on a queue when a conversation needs one.

## Prerequisites

* An agent you already have running, with an active production prompt.
  [Build a WhatsApp and SMS AI Agent](/guides/whatsapp-sms-ai-agent) covers
  the messaging form end to end.
* An API key with `agents:read` / `agents:write` scope.
* The agent's `version`-aware runtime — canary, regression, and comparison
  endpoints all operate on saved versions, so read the agent
  [versions surface](/agents/agent-versions) first.

## 1. Choose the models — chat vs voice vs retrieval

A single agent seldom uses one model for everything. A chat surface
tolerates a long reasoning model; voice needs one with a fast first-token;
retrieval benefits from a cheaper reranker. Pick per surface.

* **Chat (SMS / WhatsApp / widget).** `POST /agents` accepts any Claude
  model. [WhatsApp and SMS AI Agent](/guides/whatsapp-sms-ai-agent) covers
  the full build.
* **Voice.** First-token latency matters more than depth. The
  [voice handback guide](/voice/ai-agent-handback) is the pattern to copy,
  and it still applies here.
* **Retrieval.** Where you point `search_knowledge` does most of the
  answering. Tune it with the retrieval benchmark below.

## 2. Load knowledge sources

The prompt is one input. The knowledge base is a second — bigger and more
volatile.

```bash theme={null}
curl -X POST https://api.orbit.devotel.io/api/v1/knowledge-bases \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{ "name": "Support Docs" }'
```

```bash theme={null}
curl -X POST https://api.orbit.devotel.io/api/v1/agents/<agent-id> \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{ "knowledge_base_ids": ["kb_abc123"] }'
```

Before you ship a prompt change at all, run the retrieval benchmark over
your golden question set:

* `POST /agents/:id/retrieval-benchmark/runs` — score its recall on the
  query set you pin as baseline.
* `GET /agents/:id/retrieval-benchmark` — read the latest persisted run.

If retrieval is off, no prompt tweak will fix it. Tune knowledge sources
first.

## 3. Define guardrails

The policy decides what the agent must and must not say, regardless of
prompt.

* `GET /guardrail-policies` — list the built-in library.
* `POST /agents/:id/guardrail-policy` — apply a named policy to the agent.
* `GET /agents/guardrail-analytics` — see what the policy is firing on
  in practice, by rule and by conversation.

Guardrails run on every live turn, so define them once, before you write
the candidate prompt, and keep them attached during the rollout.

## 4. Test offline

Never let a new prompt meet a live channel first.

### Persona simulation

`POST /agents/:id/persona-simulation` replays a scripted customer persona
against the candidate prompt and grades the whole conversation against a
rubric: pass rate, mean score, per-tool calls, per-scenario verdict.

```bash theme={null}
curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/persona-simulation \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "version_id": "ver_candidate_1",
    "scenarios": [
      { "name": "refund request", "objective": "handle refund per policy" }
    ]
  }'
```

A scenario that fails here never reaches traffic. A scenario that passes
moves on. The endpoint is rate-limited tighter than a normal run (the
persona + runtime + judge fan out per scenario) — use a short, curated
set, not a dump of the whole corpus.

### Shadow comparison

`POST /agents/:id/shadow` fires a side-channel comparison run, answering
the same conversation with both the production and the candidate prompt,
without exposing the candidate to the customer.

* `GET /agents/:id/shadow-comparison?days=7` — side-by-side review of
  what the candidate would have said.
* `POST /agents/:id/shadow/promote` — take the shadow candidate directly
  into production if it wins on both quality and cost.

Only promote from shadow when the new prompt behaves better on the
conversations that matter — and only after persona simulation passed.

## 5. Roll out gradually

`POST /agents/:id/canary-rollout` starts a staged rollout for the
candidate version. The default ladder is 5% → 25% → 100%; override the
per-stage array to stay in your budget/error tolerance.

```bash theme={null}
curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/canary-rollout \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "candidate_version_id": "ver_candidate_1",
    "baseline_version_id": "ver_current_prod",
    "stages": [5, 25, 100],
    "min_stage_sample_size": 50,
    "rollback_on_warn": false
  }'
```

The persisted state lives under your organization settings — nothing new
in the schema. Three operations close the loop:

* `GET /agents/:id/canary-rollout` — read the current stage and its
  live scorecard verdict.
* `POST /agents/:id/canary-rollout/evaluate` — compare the running
  scorecard against the stage gates; the decision (advance, hold,
  rollback, complete) is applied and persisted.
* `POST /agents/:id/canary-rollout/rollback` — operator kill-switch to
  pull the candidate regardless of gates.
* `POST /agents/:id/canary-rollout/cancel` — void a rollout started by
  mistake.

## 6. Verify with regression tests

`POST /agents/:id/regression-tests` saves replayable conversations and
`POST /agents/:id/regression-tests/run-all` runs them after each deploy.

Save the hard cases. A test is a scripted conversation with expected
outputs; the run-all endpoint replays every saved test against the
candidate and reports pass/fail per test. If any of the saved tests miss
production gates, the rollout should halt or roll back — and the canary
policy knows that.

## 7. Monitor in production

Three surfaces together tell you how the live rollout is doing:

* **Quality scorecard.** `GET /agents/:agentId/quality-scorecard?window=7&min_sample=20`
  computes the resolved rate, latency, cost, and customer-sentiment index
  from the conversation-outcomes ledger.
* **Fairness evaluation.** `GET /agents/:id/fairness-eval/pack` lists the
  built-in paired-testing probes; `POST /agents/:id/fairness-eval/run`
  replays them against a chosen attribute, flagging biased tracks before
  the rollout reaches 100%.
* **Loop detection.** `POST /agents/conversations/:conversationId/loop-check`
  answers whether the agent is stuck repeating itself on a live
  conversation.
* **Human oversight.** `GET /agents/oversight/sessions` lists live voice /
  chat with an oversight state; supervisor takeover and release endpoints
  bind a human to a problematic conversation.

Pipeline the scorecard once per stage, parity once before completion —
the loop detector and oversight console run on demand.

## 8. Route to a human fallback

Some conversations should not stay with the agent at all.

* `POST /conversations/:id/handoff` — escalate the current chat
  conversation to a support queue.
* `GET /agents/crm-tools/internal/active-providers` — list the CRM providers
  your account can dispatch to.
* `POST /agents/crm-tools/internal/dispatch` — dispatch a ticketing or
  CRM tool call without leaving the agent runtime.

The same handoff target appears at the end of the WhatsApp agent guide
because it is the safe destination for anything the confidence score or
guardrails block.

## Worked example — promoting a WhatsApp support agent from 5% to 100%

You built the agent in WhatsApp/SMS form and want a more specific prompt
that deflects refunds without sacrificing tone.

1. Save the candidate prompt as a version (`PUT /agents/:id` with the new
   `system_prompt`, or `POST /agents/:id/versions` if you pin by id).
2. Run `POST /agents/:id/persona-simulation` on the refund scenario set —
   all pass or you stop here.
3. Run `POST /agents/:id/shadow` against recent conversations and review
   `GET /agents/:id/shadow-comparison?days=7` for tone regressions.
4. Start the rollout: `POST /agents/:id/canary-rollout` with
   `stages: [5, 25, 100]`.
5. After the first tick, read `GET /agents/:id/canary-rollout` and run
   `POST /agents/:id/canary-rollout/evaluate`. A decision of
   `advance` tells you the 5% stage's gates passed; hold means more
   sample is needed; rollback means the gates failed and the candidate is
   off live.
6. Run `POST /agents/:id/regression-tests/run-all` at each stage; the
   candidate must pass the saved gate set before advancing.
7. Watch `GET /agents/:agentId/quality-scorecard` and
   `POST /agents/:id/fairness-eval/run` at each stage.
8. When the evaluator reports `complete`, the candidate is serving 100%.
   If any tick returns `rollback`, the server pinned the baseline back;
   `POST rollback` performs the same pull manually.

Each numeric path in the example is the live route of that subsystem —
nothing is staged manually once the rollout exists.

## Troubleshooting

| Symptom                                               | Fix                                                                                                                                 |
| ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `422` on canary start                                 | `candidate_version_id` and `baseline_version_id` must reference saved versions; save both first.                                    |
| `hold` at a stage longer than expected                | Bump `min_stage_sample_size` or the seed sample; the gate needs enough conversations before it trusts the score.                    |
| Shadow never produced output for a given conversation | The shadow runner is on a sampled rate; check the comparison `days` window and that the base conversation itself ran to completion. |
| Regression run-all times out                          | More than \~30 saved tests; split them by domain or drop duplicates.                                                                |
| Guardrail policy blocked a needed tool                | Tune the rule on `GET /guardrail-policies/:policyId`; never suspend the whole policy.                                               |
| Scorecard shows a cost jump at a stage                | Inspect the shadow's `cost_delta` vs production before promoting; the candidate may simply be more expensive per turn.              |

## See also

* [Build a WhatsApp and SMS AI Agent](/guides/whatsapp-sms-ai-agent) — the
  messaging form this pipeline rolls out.
* [AI Voice Agent Handback](/voice/ai-agent-handback) — return an
  escalated voice call to the agent after a human step.
* [Creating Agents](/agents/creating-agents) — full agent configuration
  reference, including guardrails and escalation triggers.
* [Agent versions](/agents/agent-versions) — saved prompt configurations
  these rollout endpoints move between.
* [Handoff targets](/agents/handoff-targets) — the destination
  conversations route to when the agent hands off.