Pick a testing harness
Orbit ships six pre-deploy testing surfaces for an AI agent. They all run in sandbox mode — no outbound voice or SMS, no billing side effects — but they each answer a different question. Use this table to pick the right one for the run you are about to make, then follow its link for the request shape.How to choose
Debugging one broken flow — pick dry-run. You write the call’s turns by hand and read the trace turn by turn. Add an assertion (expected_tool_calls,
must_contain, must_not_contain) when you want the run to gate on the
behaviour, not just show it. See the Testing section of
Create and test an agent.
Gating a batch of flows before promote — pick batch-simulation. One
call replays many scripted journeys and rolls them up into a pass/fail across
the suite; wire it into CI so a prompt change cannot land green while one
journey regresses. See
A/B prompt experiments for the suite
shape.
Voice flows, STT fidelity, barge-in — pick voice-simulation. The
scripted scenarios exercise the full voice path (text-to-speech prompts in,
STT back out, optional DTMF and barge-in) and report audio-fidelity metrics
alongside behaviour assertions — per-scenario transcripts plus a suite-level
launch-gate rollup. Read runs from the dashboard in
Voice eval runs.
Load, cost, degradation under a spike — pick soak-test. One script,
many concurrent sessions, a staged ramp. It returns the per-turn latency
percentiles and the per-stage degradation curve, so you can answer “will the
agent hold p95 during a campaign?” before you promote. Pair it with the
production sampling surface for
post-launch evidence.
Safety and adversarial posture — pick red-team. A built-in pack of
jailbreak, prompt-injection-via-tool-output, data-exfiltration, and
policy-guardrail probes replays against the agent and scores each category;
the promotion gate compares the overall score against the minimum you set.
See the red-team step in
Agent from prompt.
Live-caller coverage and regression gates — pick persona-simulation.
Two modes share the endpoint: an LLM caller improvises turn-by-turn (explore
how the agent handles a mood or an unexpected customer), or a fixed
scripted_utterances script replays exactly the same conversation on every
run while the LLM judge still grades it (regression). The second mode gives a
deterministic caller plus a judge verdict — what a CI regression gate needs.
Both the persona and the scripted modes live in
Simulation & Regression Eval Suite, and
agent versions shows how the gate wires into a
promotion.
When to combine them
Run dry-run while you author the flow, promote through batch-simulation or voice-simulation depending on channel, clear the red-team gate, then hold persona-simulation as your standing regression gate. Once live, continuous production evals sample real calls back into a golden set so the judge keeps scoring on production traffic. Every harness runs in sandbox mode, so nothing here dispatches an outbound voice call or SMS — mutative tools are short-circuited, billing is skipped, and memory writes are suppressed (persona-simulation additionally pinssandboxMode internally). LLM cost applies where an LLM turn runs — the
dry-run, batch-simulation, and voice-simulation harnesses replay scripted
inputs but still call the model for the agent’s own turns.