​

Simulation & Regression Eval Suite

• Live-persona run (POST /api/v1/agents/:id/persona-simulation, the Agent → Testing → Persona panel) — an LLM caller improvises turn-by-turn. Best for exploring how the agent copes with a mood or an unexpected customer.
• Scripted regression gate (POST /api/v1/agents/:id/simulation-eval) — a fixed caller utterance script replays exactly the same conversation on every run, the LLM judge grades the finished transcript, and the batch returns a boolean passed gate. This is the mode to wire into CI: the caller text never changes, so two runs of the same suite compare cleanly.

​

1. What the suite does

1. Drives a live LLM caller turn-by-turn against your agent (not a scripted prompt list).
2. Sends the full transcript to the LLM judge, which scores it against the objective and each rubric criterion.
3. Returns a rollout-gate report: pass rate, mean judge score, per-scenario verdicts, and the tools the agent invoked.

​

2. What goes into a scenario

​

3. Run the suite

curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/persona-simulation \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "scenarios": [
      {
        "id": "refund-request",
        "title": "Refund request",
        "persona": { "name": "Dana", "mood": "frustrated" },
        "situation": "Charged twice for the same invoice.",
        "objective": "Refund the duplicate charge with a clear timeline.",
        "rubric": [
          { "key": "empathy", "label": "Acknowledges the frustration", "description": "Opens with empathy, not a canned greeting." },
          { "key": "policy", "label": "Follows the refund policy", "description": "Pays the duplicate back within plan rules, and says so clearly." }
        ],
        "successCriteria": "Conversation ends with a refund initiated or a human escalation offered."
      }
    ],
    "threshold": 75
  }'

​

Scripted regression gate (simulation-eval)

curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/simulation-eval \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "scenarios": [
      {
        "id": "refund-request",
        "title": "Refund request",
        "scripted_utterances": ["You charged me twice.", "It is order 42.", "Please refund it."],
        "situation": "Charged twice for the same invoice.",
        "objective": "Refund the duplicate charge with a clear timeline.",
        "success_criteria": "Conversation ends with a refund initiated or a human escalation offered."
      }
    ],
    "min_pass_rate": 1,
    "max_error_rate": 0
  }'

​

4. Common results and what to do about them

• A 503 “LLM_UNAVAILABLE” envelope — the LLM provider is not configured for the environment. The suite cannot run until that is fixed by an owner/admin.
• A 422 “VALIDATION_ERROR” envelope — one of the field constraints (empty title / persona.name / situation / objective, or more than 10 rubric criteria) is violated. Fix the scenario and retry.
• A 404 envelope from the scripted gate — the agent id does not exist in this tenant; the gate checks for the agent before spending a single runtime call.
• A scenario fails on rubric criteria — inspect the per-scenario judge reasoning in the panel (or the reasoning text in the API response) before promoting the candidate prompt/model.
• The whole suite times out — the endpoint is rate-limited tighter than ordinary calls (the persona + runtime + judge fan out per scenario). Split the batch and retry.

​

5. Where it fits in the rollout pipeline

1. Replay regression tests and run persona simulations against the candidate (guide: Safely Roll Out an AI Agent).
2. Promote the version (docs: Agent Versions).
3. Keep production sampling enabled (docs: Continuous Production Evals).