> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Persona-simulation harness — live callers vs scripted regression mode

> Drive an LLM-simulated caller persona against your agent end-to-end and grade the finished transcript with an LLM judge — or replay a fixed caller script deterministically as a regression gate on every prompt change.

# Persona-simulation harness

`POST /api/v1/agents/:id/persona-simulation` is the flagship pre-deploy test
harness for an AI agent. It runs a full conversation — caller turns, agent
turns, tool calls — then has an LLM judge grade the finished transcript
against your scenario's objective and rubric, and returns a rollout-gate
report you can use to block a bad prompt or model change before it reaches
customers.

Endpoint paths below are relative; send them against
`https://api.orbit.devotel.io`.

## How it differs from the other harnesses

Orbit ships six pre-deploy testers. Pick the persona-simulation harness when
the question is "how does the agent behave in a real conversation" rather
than "did the reply contain the right string":

| Harness                                                | What it exercises                           | Where it fits                                        |
| ------------------------------------------------------ | ------------------------------------------- | ---------------------------------------------------- |
| **Dry run**                                            | One scripted scenario, turn-by-turn trace   | Quick smoke check of a single flow                   |
| **Batch simulation** (`POST .../batch-simulation`)     | A suite of scripted text journeys           | Scripted text coverage with deterministic assertions |
| **Voice simulation** (`POST .../voice-simulation`)     | A suite of scripted voice journeys          | The same, over the voice path                        |
| **Soak test** (`POST .../soak-test`)                   | A ramp of concurrent scripted sessions      | Load shape, not response quality                     |
| **Red team** (`POST .../red-team`)                     | Adversarial scripted probes                 | Safety and jailbreak resistance                      |
| **Persona simulation** (`POST .../persona-simulation`) | A live or scripted caller, LLM-judge graded | Behavioural quality gate before rollout              |

The first five replay caller turns an operator wrote by hand — the "persona"
is a label on a fixed script, and assertions are deterministic at most
(contains/excludes, tool-called). Persona simulation grades the whole
conversation with an LLM judge, which is what makes the verdict meaningful
for a prompt, model, or config change.

Every mode of this harness runs against the agent's sandboxed runtime —
no billing, no memory writes, no outbound calls or messages ever leave the
platform.

## Live-persona mode

Omit `scripted_utterances` on a scenario and an LLM-simulated caller persona
drives the conversation. The persona reacts turn-by-turn to what the agent
actually says — it improvises within the mood, background, and situation you
gave it — until the caller reports the issue resolved or the turn cap is
reached. Then the LLM judge scores the finished transcript.

This is the exploratory mode: use it to find out how the agent copes with an
angry customer, a confused one, or an unexpected tangent, without writing
the conversation yourself.

```bash theme={null}
curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/persona-simulation \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "scenarios": [
      {
        "id": "angry-refund",
        "title": "Angry duplicate-charge refund",
        "channel": "chat",
        "difficulty": "hard",
        "persona": {
          "name": "Dana",
          "mood": "angry",
          "background": "Long-time customer who was charged twice for invoice INV-1042 and already tried the support form once.",
          "style": "impatient, terse"
        },
        "situation": "Dana sees two identical charges on the card statement and demands an immediate refund.",
        "objective": "Verify the duplicate charge, initiate the refund within policy, and de-escalate without transferring unless Dana asks.",
        "rubric": [
          {
            "key": "deescalation",
            "label": "De-escalates the anger",
            "description": "Acknowledges the frustration in the first reply; never dismisses or blames."
          },
          {
            "key": "policy_adherence",
            "label": "Stays inside refund policy",
            "description": "Refunds a verified duplicate charge; does not promise exceptions plan rules do not allow."
          }
        ],
        "successCriteria": "The conversation ends with a refund initiated or a clear, policy-backed explanation of the next step.",
        "max_turns": 8
      }
    ],
    "threshold": 80,
    "concurrency": 2
  }'
```

Respond to the verdict, not the transcript length: a long conversation is
not a pass. The judge scores the outcome against your objective and each
rubric criterion.

## Scripted regression mode

Add `scripted_utterances` — one caller utterance per turn — and the run
becomes deterministic. The exact same caller script is replayed verbatim on
every run regardless of what the agent says; only the agent's replies (and
the judge's verdict) can move. That makes it the regression gate for prompt,
model, and config changes: the same conversation replays against the
candidate, and the LLM judge grades it, so a score diff tracks the change
you made instead of a different conversation.

```bash theme={null}
curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/persona-simulation \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "scenarios": [
      {
        "id": "angry-refund-regression",
        "title": "Regression: duplicate-charge refund",
        "persona": { "name": "Dana", "mood": "angry" },
        "scripted_utterances": [
          "You charged my card twice for the same invoice. Fix it now.",
          "It is invoice INV-1042, charged on the 3rd and again on the 4th.",
          "I do not care about process. I want the money back today.",
          "Fine. As long as the refund is actually started, that works."
        ],
        "situation": "Dana demands a refund for a verified duplicate charge.",
        "objective": "Verify the duplicate charge, initiate the refund within policy, and stay calm under pressure.",
        "rubric": [
          {
            "key": "deescalation",
            "label": "De-escalates the anger",
            "description": "Acknowledges the frustration early; never argues with the customer."
          },
          {
            "key": "policy_adherence",
            "label": "Stays inside refund policy",
            "description": "Refunds the verified duplicate; does not promise same-day settlement a policy cannot guarantee."
          }
        ]
      }
    ],
    "threshold": 80,
    "judge_model": "claude-haiku-4-5-20251001"
  }'
```

Two practices keep the gate honest over time:

* **Pin `judge_model`** in regression suites. The judge is an LLM too —
  pinning the model means a verdict diff between two runs reflects your
  agent change, not judge drift. The value must come from the supported
  Claude model list; a 422 otherwise.
* **Reuse the same scenario `id`s.** Verdicts are reported per scenario id,
  so stable ids give you a clean pass/fail diff between a baseline and a
  candidate run.

When you need a saved, nameable suite with an explicit boolean gate
(`min_pass_rate`, `max_error_rate`) instead of building the array per call,
the scripted-regression endpoint (`POST .../agents/:id/simulation-eval`)
carries that shape — see
[Simulation & Regression Eval Suite](/agents/simulation-eval-suite). The
persona simulation route accepts a mixed batch: scenarios with
`scripted_utterances` replay deterministically, the rest run as live
personas, and the response's `replay_modes` block reports the mix.

## Pass/fail thresholds and blocking a rollout

Each run returns a batch summary plus a per-scenario verdict:

* `summary.passRate` — share of scenarios that passed (errored scenarios
  count against it).
* `summary.avgScore` — mean judge score (0–100) across scored scenarios.
* `summary.totalCostCents` — what the batch cost to run.
* Per scenario: `status` (`pass` / `fail` / `error`), `score`, `passed`,
  the judge's `reasoning`, turn count, and the tools the agent invoked.

`threshold` (0–100, default 70) sets the judge's pass line for every
scenario in the batch. A scenario passes when its judge score meets the
threshold.

To block a rollout on this harness:

1. Save a version of the agent
   ([Agent versions](/agents/agent-versions)) — the candidate you intend to
   promote.
2. Run the harness against it with `candidate_version_id`. Omit that field
   and the live config is simulated, which is useful for a baseline but not
   for a gate.
3. In CI, treat a scenario with `status: "fail"` (or `passed: false` in the
   raw report) as a blocking signal on the candidate version; read the
   judge's `reasoning` for why. Fail closed on `error` — an errored scenario
   is unscored, not a pass.
4. Only if the gate clears, promote the version. When the gate fails, the
   candidate never meets traffic; fix the prompt or model and re-run the
   same scenario ids.

The endpoint is restricted to owner, admin, and developer roles and is
rate-limited tighter than ordinary calls (2 requests/minute) — every
scenario turn fans out to a persona call, a sandboxed runtime hop, and a
judge call. Send a short curated scenario set (your hardest cases), not the
full test corpus; a single batch is capped at 100 scenarios and each
scenario at 12 turns.

## Failure modes to expect

* **503 `LLM_UNAVAILABLE`** — the LLM judge is not configured on this
  deployment; the harness cannot run until an owner configures it.
* **404 `NOT_FOUND`** — the agent id does not exist in your workspace. The
  check runs before any simulation spend.
* **422 `VALIDATION_ERROR`** — a field constraint was violated: scenario
  fields like `id`, `title`, `persona.name`, `situation`, and `objective`
  are required; `rubric` holds at most 10 criteria; `scripted_utterances`
  holds one non-empty utterance per turn, up to the 12-turn cap.
* **502 `AI_ERROR`** — the batch failed mid-run upstream; retry, and split
  large batches if it repeats.
* **Scenario-level `status: "error"`** — that one conversation or its judge
  call failed; other scenarios in the batch still report normally. The
  scenario's `error` field says what happened.

## Where it fits in the pipeline

Persona simulation is the offline behavioural gate in the middle of the
rollout ladder:

1. Run persona simulations (and replay scripted regression scenarios)
   against the candidate version.
2. Promote the version and stage it on the
   [canary ladder](/agents/agent-versions).
3. Keep production sampling on with
   [continuous production evals](/agents/continuous-production-evals).

A passing batch does not replace the canary — it is the fast, cheap screen
that rejects a bad candidate before it touches a single customer. For the
end-to-end release path, see
[Safely roll out an AI agent](/guides/ai-agent-rollout-pipeline).