Skip to main content

​
Persona-simulation harness

POST /api/v1/agents/:id/persona-simulation is the flagship pre-deploy test harness for an AI agent. It runs a full conversation — caller turns, agent turns, tool calls — then has an LLM judge grade the finished transcript against your scenario’s objective and rubric, and returns a rollout-gate report you can use to block a bad prompt or model change before it reaches customers. Endpoint paths below are relative; send them against https://api.orbit.devotel.io.

​
How it differs from the other harnesses

Orbit ships six pre-deploy testers. Pick the persona-simulation harness when the question is “how does the agent behave in a real conversation” rather than “did the reply contain the right string”: The first five replay caller turns an operator wrote by hand — the “persona” is a label on a fixed script, and assertions are deterministic at most (contains/excludes, tool-called). Persona simulation grades the whole conversation with an LLM judge, which is what makes the verdict meaningful for a prompt, model, or config change. Every mode of this harness runs against the agent’s sandboxed runtime — no billing, no memory writes, no outbound calls or messages ever leave the platform.

​
Live-persona mode

Omit scripted_utterances on a scenario and an LLM-simulated caller persona drives the conversation. The persona reacts turn-by-turn to what the agent actually says — it improvises within the mood, background, and situation you gave it — until the caller reports the issue resolved or the turn cap is reached. Then the LLM judge scores the finished transcript. This is the exploratory mode: use it to find out how the agent copes with an angry customer, a confused one, or an unexpected tangent, without writing the conversation yourself.
Respond to the verdict, not the transcript length: a long conversation is not a pass. The judge scores the outcome against your objective and each rubric criterion.

​
Scripted regression mode

Add scripted_utterances — one caller utterance per turn — and the run becomes deterministic. The exact same caller script is replayed verbatim on every run regardless of what the agent says; only the agent’s replies (and the judge’s verdict) can move. That makes it the regression gate for prompt, model, and config changes: the same conversation replays against the candidate, and the LLM judge grades it, so a score diff tracks the change you made instead of a different conversation.
Two practices keep the gate honest over time:
  • Pin judge_model in regression suites. The judge is an LLM too — pinning the model means a verdict diff between two runs reflects your agent change, not judge drift. The value must come from the supported Claude model list; a 422 otherwise.
  • Reuse the same scenario ids. Verdicts are reported per scenario id, so stable ids give you a clean pass/fail diff between a baseline and a candidate run.
When you need a saved, nameable suite with an explicit boolean gate (min_pass_rate, max_error_rate) instead of building the array per call, the scripted-regression endpoint (POST .../agents/:id/simulation-eval) carries that shape — see Simulation & Regression Eval Suite. The persona simulation route accepts a mixed batch: scenarios with scripted_utterances replay deterministically, the rest run as live personas, and the response’s replay_modes block reports the mix.

​
Pass/fail thresholds and blocking a rollout

Each run returns a batch summary plus a per-scenario verdict:
  • summary.passRate — share of scenarios that passed (errored scenarios count against it).
  • summary.avgScore — mean judge score (0–100) across scored scenarios.
  • summary.totalCostCents — what the batch cost to run.
  • Per scenario: status (pass / fail / error), score, passed, the judge’s reasoning, turn count, and the tools the agent invoked.
threshold (0–100, default 70) sets the judge’s pass line for every scenario in the batch. A scenario passes when its judge score meets the threshold. To block a rollout on this harness:
  1. Save a version of the agent (Agent versions) — the candidate you intend to promote.
  2. Run the harness against it with candidate_version_id. Omit that field and the live config is simulated, which is useful for a baseline but not for a gate.
  3. In CI, treat a scenario with status: "fail" (or passed: false in the raw report) as a blocking signal on the candidate version; read the judge’s reasoning for why. Fail closed on error — an errored scenario is unscored, not a pass.
  4. Only if the gate clears, promote the version. When the gate fails, the candidate never meets traffic; fix the prompt or model and re-run the same scenario ids.
The endpoint is restricted to owner, admin, and developer roles and is rate-limited tighter than ordinary calls (2 requests/minute) — every scenario turn fans out to a persona call, a sandboxed runtime hop, and a judge call. Send a short curated scenario set (your hardest cases), not the full test corpus; a single batch is capped at 100 scenarios and each scenario at 12 turns.

​
Failure modes to expect

  • 503 LLM_UNAVAILABLE — the LLM judge is not configured on this deployment; the harness cannot run until an owner configures it.
  • 404 NOT_FOUND — the agent id does not exist in your workspace. The check runs before any simulation spend.
  • 422 VALIDATION_ERROR — a field constraint was violated: scenario fields like id, title, persona.name, situation, and objective are required; rubric holds at most 10 criteria; scripted_utterances holds one non-empty utterance per turn, up to the 12-turn cap.
  • 502 AI_ERROR — the batch failed mid-run upstream; retry, and split large batches if it repeats.
  • Scenario-level status: "error" — that one conversation or its judge call failed; other scenarios in the batch still report normally. The scenario’s error field says what happened.

​
Where it fits in the pipeline

Persona simulation is the offline behavioural gate in the middle of the rollout ladder:
  1. Run persona simulations (and replay scripted regression scenarios) against the candidate version.
  2. Promote the version and stage it on the canary ladder.
  3. Keep production sampling on with continuous production evals.
A passing batch does not replace the canary — it is the fast, cheap screen that rejects a bad candidate before it touches a single customer. For the end-to-end release path, see Safely roll out an AI agent.