Safely Roll Out an AI Agent

Prompt and model changes are production changes. The same discipline you apply to a code deploy — test offline, inspect against a stop list, roll a slice of traffic, watch the quality numbers, keep a one-call rollback — applies to an agent’s system_prompt, model, or knowledge-base update. Orbit already ships each stage; this guide wires them into one pipeline. The endpoint paths below are relative. Send them against https://api.orbit.devotel.io/api/v1.

What the pipeline covers

Choosing a model/runtime profile (chat vs voice vs retrieval).
Grounding the agent in your knowledge sources, before launch.
Defining the guardrail policy that shapes what can be said.
Offline comparison of the candidate against the prompt in production.
Graduated rollout that shifts one small traffic slice at a time.
Automatic regression runs on every publish.
Live scoring, drift detection, and a human-in-the-loop when you want it.
A clean fallback to a person on a queue when a conversation needs one.

Prerequisites

An agent you already have running, with an active production prompt. Build a WhatsApp and SMS AI Agent covers the messaging form end to end.
An API key with agents:read / agents:write scope.
The agent’s version-aware runtime — canary, regression, and comparison endpoints all operate on saved versions, so read the agent versions surface first.

1. Choose the models — chat vs voice vs retrieval

A single agent seldom uses one model for everything. A chat surface tolerates a long reasoning model; voice needs one with a fast first-token; retrieval benefits from a cheaper reranker. Pick per surface.

Chat (SMS / WhatsApp / widget). POST /agents accepts any Claude model. WhatsApp and SMS AI Agent covers the full build.
Voice. First-token latency matters more than depth. The voice handback guide is the pattern to copy, and it still applies here.
Retrieval. Where you point search_knowledge does most of the answering. Tune it with the retrieval benchmark below.

2. Load knowledge sources

The prompt is one input. The knowledge base is a second — bigger and more volatile.

curl -X POST https://api.orbit.devotel.io/api/v1/knowledge-bases \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{ "name": "Support Docs" }'

curl -X POST https://api.orbit.devotel.io/api/v1/agents/<agent-id> \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{ "knowledge_base_ids": ["kb_abc123"] }'

Before you ship a prompt change at all, run the retrieval benchmark over your golden question set:

POST /agents/:id/retrieval-benchmark/runs — score its recall on the query set you pin as baseline.
GET /agents/:id/retrieval-benchmark — read the latest persisted run.

If retrieval is off, no prompt tweak will fix it. Tune knowledge sources first.

3. Define guardrails

The policy decides what the agent must and must not say, regardless of prompt.

GET /guardrail-policies — list the built-in library.
POST /agents/:id/guardrail-policy — apply a named policy to the agent.
GET /agents/guardrail-analytics — see what the policy is firing on in practice, by rule and by conversation.

Guardrails run on every live turn, so define them once, before you write the candidate prompt, and keep them attached during the rollout.

4. Test offline

Never let a new prompt meet a live channel first.

Persona simulation

POST /agents/:id/persona-simulation replays a scripted customer persona against the candidate prompt and grades the whole conversation against a rubric: pass rate, mean score, per-tool calls, per-scenario verdict.

curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/persona-simulation \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "version_id": "ver_candidate_1",
    "scenarios": [
      { "name": "refund request", "objective": "handle refund per policy" }
    ]
  }'

A scenario that fails here never reaches traffic. A scenario that passes moves on. The endpoint is rate-limited tighter than a normal run (the persona + runtime + judge fan out per scenario) — use a short, curated set, not a dump of the whole corpus.

Shadow comparison

POST /agents/:id/shadow fires a side-channel comparison run, answering the same conversation with both the production and the candidate prompt, without exposing the candidate to the customer.

GET /agents/:id/shadow-comparison?days=7 — side-by-side review of what the candidate would have said.
POST /agents/:id/shadow/promote — take the shadow candidate directly into production if it wins on both quality and cost.

Only promote from shadow when the new prompt behaves better on the conversations that matter — and only after persona simulation passed.

5. Roll out gradually

POST /agents/:id/canary-rollout starts a staged rollout for the candidate version. The default ladder is 5% → 25% → 100%; override the per-stage array to stay in your budget/error tolerance.

curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/canary-rollout \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "candidate_version_id": "ver_candidate_1",
    "baseline_version_id": "ver_current_prod",
    "stages": [5, 25, 100],
    "min_stage_sample_size": 50,
    "rollback_on_warn": false
  }'

The persisted state lives under your organization settings — nothing new in the schema. Three operations close the loop:

GET /agents/:id/canary-rollout — read the current stage and its live scorecard verdict.
POST /agents/:id/canary-rollout/evaluate — compare the running scorecard against the stage gates; the decision (advance, hold, rollback, complete) is applied and persisted.
POST /agents/:id/canary-rollout/rollback — operator kill-switch to pull the candidate regardless of gates.
POST /agents/:id/canary-rollout/cancel — void a rollout started by mistake.

6. Verify with regression tests

POST /agents/:id/regression-tests saves replayable conversations and POST /agents/:id/regression-tests/run-all runs them after each deploy. Save the hard cases. A test is a scripted conversation with expected outputs; the run-all endpoint replays every saved test against the candidate and reports pass/fail per test. If any of the saved tests miss production gates, the rollout should halt or roll back — and the canary policy knows that.

7. Monitor in production

Three surfaces together tell you how the live rollout is doing:

Quality scorecard. GET /agents/:agentId/quality-scorecard?window=7&min_sample=20 computes the resolved rate, latency, cost, and customer-sentiment index from the conversation-outcomes ledger.
Fairness evaluation. GET /agents/:id/fairness-eval/pack lists the built-in paired-testing probes; POST /agents/:id/fairness-eval/run replays them against a chosen attribute, flagging biased tracks before the rollout reaches 100%.
Loop detection. POST /agents/conversations/:conversationId/loop-check answers whether the agent is stuck repeating itself on a live conversation.
Human oversight. GET /agents/oversight/sessions lists live voice / chat with an oversight state; supervisor takeover and release endpoints bind a human to a problematic conversation.

Pipeline the scorecard once per stage, parity once before completion — the loop detector and oversight console run on demand.

8. Route to a human fallback

Some conversations should not stay with the agent at all.

POST /conversations/:id/handoff — escalate the current chat conversation to a support queue.
GET /agents/crm-tools/internal/active-providers — list the CRM providers your account can dispatch to.
POST /agents/crm-tools/internal/dispatch — dispatch a ticketing or CRM tool call without leaving the agent runtime.

The same handoff target appears at the end of the WhatsApp agent guide because it is the safe destination for anything the confidence score or guardrails block.

Worked example — promoting a WhatsApp support agent from 5% to 100%

You built the agent in WhatsApp/SMS form and want a more specific prompt that deflects refunds without sacrificing tone.

Save the candidate prompt as a version (PUT /agents/:id with the new system_prompt, or POST /agents/:id/versions if you pin by id).
Run POST /agents/:id/persona-simulation on the refund scenario set — all pass or you stop here.
Run POST /agents/:id/shadow against recent conversations and review GET /agents/:id/shadow-comparison?days=7 for tone regressions.
Start the rollout: POST /agents/:id/canary-rollout with stages: [5, 25, 100].
After the first tick, read GET /agents/:id/canary-rollout and run POST /agents/:id/canary-rollout/evaluate. A decision of advance tells you the 5% stage’s gates passed; hold means more sample is needed; rollback means the gates failed and the candidate is off live.
Run POST /agents/:id/regression-tests/run-all at each stage; the candidate must pass the saved gate set before advancing.
Watch GET /agents/:agentId/quality-scorecard and POST /agents/:id/fairness-eval/run at each stage.
When the evaluator reports complete, the candidate is serving 100%. If any tick returns rollback, the server pinned the baseline back; POST rollback performs the same pull manually.

Each numeric path in the example is the live route of that subsystem — nothing is staged manually once the rollout exists.

Troubleshooting

Symptom	Fix
`422` on canary start	`candidate_version_id` and `baseline_version_id` must reference saved versions; save both first.
`hold` at a stage longer than expected	Bump `min_stage_sample_size` or the seed sample; the gate needs enough conversations before it trusts the score.
Shadow never produced output for a given conversation	The shadow runner is on a sampled rate; check the comparison `days` window and that the base conversation itself ran to completion.
Regression run-all times out	More than ~30 saved tests; split them by domain or drop duplicates.
Guardrail policy blocked a needed tool	Tune the rule on `GET /guardrail-policies/:policyId`; never suspend the whole policy.
Scorecard shows a cost jump at a stage	Inspect the shadow’s `cost_delta` vs production before promoting; the candidate may simply be more expensive per turn.

WhatsApp

Guides

Messaging Controls

Audience & Support Tools

Safely Roll Out an AI Agent

Safely Roll Out an AI Agent

What the pipeline covers

Prerequisites

1. Choose the models — chat vs voice vs retrieval

2. Load knowledge sources

3. Define guardrails

4. Test offline

Persona simulation

Shadow comparison

5. Roll out gradually

6. Verify with regression tests

7. Monitor in production

8. Route to a human fallback

Worked example — promoting a WhatsApp support agent from 5% to 100%

Troubleshooting

See also

​Safely Roll Out an AI Agent

​What the pipeline covers

​Prerequisites

​1. Choose the models — chat vs voice vs retrieval

​2. Load knowledge sources

​3. Define guardrails

​4. Test offline

​Persona simulation

​Shadow comparison

​5. Roll out gradually

​6. Verify with regression tests

​7. Monitor in production

​8. Route to a human fallback

​Worked example — promoting a WhatsApp support agent from 5% to 100%

​Troubleshooting

​See also

Safely Roll Out an AI Agent

What the pipeline covers

Prerequisites

1. Choose the models — chat vs voice vs retrieval

2. Load knowledge sources

3. Define guardrails

4. Test offline

Persona simulation

Shadow comparison

5. Roll out gradually

6. Verify with regression tests

7. Monitor in production

8. Route to a human fallback

Worked example — promoting a WhatsApp support agent from 5% to 100%

Troubleshooting

See also