> ## Documentation Index > Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt > Use this file to discover all available pages before exploring further. # Simulation & Regression Eval Suite > Author a named suite of scripted multi-turn test scenarios with an LLM-judge rubric, then rerun it as a regression gate on every prompt or model change. # Simulation & Regression Eval Suite The **Simulation & Regression Eval Suite** runs a crafted set of customer scenarios against an agent and grades every transcript with an LLM judge — so a prompt or model change gets pass/fail evidence before it reaches a live channel. It is the pre-deploy regression gate behind the **Agent → Testing → Persona** panel and the `POST /api/v1/agents/:id/persona-simulation` endpoint. Endpoint paths below are relative; send them against `https://api.orbit.devotel.io`. The dashboard path is the same surface — **Agents → your agent → Testing → Persona**. ## 1. What the suite does Each scenario describes a customer persona (name, mood), the situation, the objective the agent must achieve, and optional rubric **criteria** the judge grades against. On every run, the simulator: 1. Drives a live LLM caller turn-by-turn against your agent (not a scripted prompt list). 2. Sends the full transcript to the LLM judge, which scores it against the objective and each rubric criterion. 3. Returns a rollout-gate report: pass rate, mean judge score, per-scenario verdicts, and the tools the agent invoked. Name the suite so the same scenarios can be rerun as a regression gate on every prompt or model change. The suite runs in sandbox mode — no billing, no memory writes, no outbound voice or SMS. ## 2. What goes into a scenario | Field | Required | Purpose | | -------------------- | -------- | ----------------------------------------------------------------------------------------------- | | `id` | yes | Stable handle for comparisons across runs. | | `title` | yes | Human-readable label. | | `persona.name` | yes | The caller identity. | | `persona.mood` | no | One of `neutral`, `happy`, `confused`, `anxious`, `frustrated`, `angry`. Defaults to `neutral`. | | `persona.background` | no | Extra context the persona simulates. | | `persona.style` | no | Voice/register hint (e.g. "impatient, terse"). | | `persona.language` | no | Override the conversation language. | | `situation` | yes | What the customer wants to accomplish. | | `objective` | yes | The behaviour the agent must achieve. | | `rubric` | no | `key` / `label` / `description` criteria the judge grades. Max 10. | | `hiddenContext` | no | Facts the persona knows but the agent does not. | | `successCriteria` | no | Extra pass condition the judge checks. | | `maxTurns` | no | Turn cap for the conversation (defaults to the global cap). | The endpoint rejects more than the supported batch size (default: all scenarios in one payload, up to the per-endpoint maximum) — send a short curated set, not a dump of the corpus. ## 3. Run the suite Dashboard path: **Agents → your agent → Testing → Persona** — enter/edit scenarios inline, set the optional candidate version, and press **Run suite**. API: ```bash theme={null} curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/persona-simulation \ -H "X-API-Key: dv_live_sk_..." \ -H "Content-Type: application/json" \ -d '{ "scenarios": [ { "id": "refund-request", "title": "Refund request", "persona": { "name": "Dana", "mood": "frustrated" }, "situation": "Charged twice for the same invoice.", "objective": "Refund the duplicate charge with a clear timeline.", "rubric": [ { "key": "empathy", "label": "Acknowledges the frustration", "description": "Opens with empathy, not a canned greeting." }, { "key": "policy", "label": "Follows the refund policy", "description": "Pays the duplicate back within plan rules, and says so clearly." } ], "successCriteria": "Conversation ends with a refund initiated or a human escalation offered." } ], "threshold": 75 }' ``` The response carries the aggregate summary (total scenarios, pass rate, mean judge score, total cost in cents, per-scenario verdicts) plus the invoked tool list. A scenario that fails never reaches traffic; one that passes moves on. ## 4. Common results and what to do about them * **A 503 "LLM\_UNAVAILABLE" envelope** — the LLM provider is not configured for the environment. The suite cannot run until that is fixed by an owner/admin. * **A 422 "VALIDATION\_ERROR" envelope** — one of the field constraints (empty `title` / `persona.name` / `situation` / `objective`, or more than 10 rubric criteria) is violated. Fix the scenario and retry. * **A scenario fails on rubric criteria** — inspect the per-scenario judge reasoning in the panel (or the `reasoning` text in the API response) before promoting the candidate prompt/model. * **The whole suite times out** — the endpoint is rate-limited tighter than ordinary calls (the persona + runtime + judge fan out per scenario). Split the batch and retry. ## 5. Where it fits in the rollout pipeline The eval suite is the offline gate that qualifies a candidate before the canary ladder accepts it: 1. Replay regression tests and run **persona simulations** against the candidate ([guide: Safely Roll Out an AI Agent](/guides/ai-agent-rollout-pipeline)). 2. Promote the version ([docs: Agent Versions](/agents/agent-versions)). 3. Keep production sampling enabled ([docs: Continuous Production Evals](/agents/continuous-production-evals)). A passing simulation suite does not replace these — it is the fast, cheap screen that rejects a bad candidate before it touches production traffic.