Skip to main content

​
Simulation & Regression Eval Suite

The Simulation & Regression Eval Suite runs a crafted set of customer scenarios against an agent and grades every transcript with an LLM judge — so a prompt or model change gets pass/fail evidence before it reaches a live channel. It is the pre-deploy regression gate behind the Agent → Testing → Persona panel and the POST /api/v1/agents/:id/persona-simulation endpoint. Endpoint paths below are relative; send them against https://api.orbit.devotel.io. The dashboard path is the same surface — Agents → your agent → Testing → Persona.

​
1. What the suite does

Each scenario describes a customer persona (name, mood), the situation, the objective the agent must achieve, and optional rubric criteria the judge grades against. On every run, the simulator:
  1. Drives a live LLM caller turn-by-turn against your agent (not a scripted prompt list).
  2. Sends the full transcript to the LLM judge, which scores it against the objective and each rubric criterion.
  3. Returns a rollout-gate report: pass rate, mean judge score, per-scenario verdicts, and the tools the agent invoked.
Name the suite so the same scenarios can be rerun as a regression gate on every prompt or model change. The suite runs in sandbox mode — no billing, no memory writes, no outbound voice or SMS.

​
2. What goes into a scenario

The endpoint rejects more than the supported batch size (default: all scenarios in one payload, up to the per-endpoint maximum) — send a short curated set, not a dump of the corpus.

​
3. Run the suite

Dashboard path: Agents → your agent → Testing → Persona — enter/edit scenarios inline, set the optional candidate version, and press Run suite. API:
The response carries the aggregate summary (total scenarios, pass rate, mean judge score, total cost in cents, per-scenario verdicts) plus the invoked tool list. A scenario that fails never reaches traffic; one that passes moves on.

​
4. Common results and what to do about them

  • A 503 “LLM_UNAVAILABLE” envelope — the LLM provider is not configured for the environment. The suite cannot run until that is fixed by an owner/admin.
  • A 422 “VALIDATION_ERROR” envelope — one of the field constraints (empty title / persona.name / situation / objective, or more than 10 rubric criteria) is violated. Fix the scenario and retry.
  • A scenario fails on rubric criteria — inspect the per-scenario judge reasoning in the panel (or the reasoning text in the API response) before promoting the candidate prompt/model.
  • The whole suite times out — the endpoint is rate-limited tighter than ordinary calls (the persona + runtime + judge fan out per scenario). Split the batch and retry.

​
5. Where it fits in the rollout pipeline

The eval suite is the offline gate that qualifies a candidate before the canary ladder accepts it:
  1. Replay regression tests and run persona simulations against the candidate (guide: Safely Roll Out an AI Agent).
  2. Promote the version (docs: Agent Versions).
  3. Keep production sampling enabled (docs: Continuous Production Evals).
A passing simulation suite does not replace these — it is the fast, cheap screen that rejects a bad candidate before it touches production traffic.