Skip to main content

​
Agent hardening runs: regression, red-team, and soak

A dataset eval scores an agent against a fixed corpus. It will not tell you whether yesterday’s prompt edit broke an answer that used to work, whether a jailbreak that was closed last month now gets through, or whether latency creeps up by turn ninety of a long conversation. Three runners cover those classes: Evals are static expected-output datasets scored by a judge; these three are mode-specific runners that drive live (sandboxed) conversations against the agent. They complement, not replace, the eval loop in Agent evals: datasets, runs, and pass-rate gates. Endpoint paths below are relative. Send them against https://api.orbit.devotel.io/api/v1. All routes need agents:read / agents:write scope and an owner, admin, or developer role. Every run executes in sandbox mode: no billing, no memory writes, and mutative tool side effects are short-circuited, so you can run all three against a live agent at any time.

​
1. Prerequisites

  • An agent id — the agent_... id from the Agents page or GET /agents.
  • A deployed baseline version — runs score a saved version. Save the live config as one version and your candidate revision as another before comparing; see Agent versions.
  • An expected-outputs dataset — regression tests use the same curation discipline as eval datasets: scripted rows for the cases you know, plus promoted production conversations for the cases you did not. Build it as described in Agent evals: datasets, runs, and pass-rate gates, then mirror its strongest rows as regression tests.

​
2. Regression tests

A regression test is a scripted conversation plus the assertions a replay is scored against. Save one per case you must never break:
expected_outputs supports contains (every string must appear), contains_any (at least one must appear), excludes (none may appear), and verbatim (exact-reply match). pinned_version_id makes the test replay against that version by default; omit it to replay against whatever version the run targets. Repeat the save call for each case — a corpus of twenty to fifty curated tests is more useful than two hundred loose ones. GET /agents/{id}/regression-tests lists the saved set with each test’s last recorded verdict; DELETE /agents/{id}/regression-tests/{testId} removes one.

​
Replay the corpus against a candidate

run-all replays every saved test and returns a per-test pass / fail / error verdict with the agent’s actual reply and the reasons any assertion failed. Pass agent_version_id to score the whole corpus against a candidate version without touching the live config:
Read the diff, not the score. Run the same corpus against the baseline version (omit agent_version_id, or pass the baseline’s id) and compare verdicts test by test:
  • No test that passed on the baseline fails on the candidate — the revision is safe to promote, regression-wise.
  • Any pass-to-fail flip — hold the promotion. The reasons array names the assertion that broke; fix the prompt, re-save the candidate as a new version, replay. Only promote when the flip list is empty or every flip is a deliberate behavior change you re-baselined.
  • error verdicts — a replay that timed out or failed at the transport layer. Treat a burst of errors as infrastructure noise and re-run before judging the candidate on them.
Replays run sequentially against the sandboxed runtime and are capped per run to keep wall-clock bounded; if your corpus is large, split it across agents’ test sets rather than fighting the cap.

​
3. Red-team runs

The red-team runner replays a built-in adversarial probe pack against the agent and returns a safety scorecard. Inspect the catalogue first:
The pack covers four categories: jailbreak, prompt_injection, data_exfiltration, and policy_guardrail. Run all of them, or narrow with a categories filter:
The response is a per-probe results array plus a rolled-up scorecard. Each probe scores resisted, compromised, or error (a probe that fails at the transport layer scores as error, never as a compromise), and the scorecard rolls the run into a safety score from 0 to 100 — resisted over resisted plus compromised:
Triage every compromised probe before promotion:
  1. Tool-mediated leaks or injected tool output — tighten the tool’s approval policy and argument allowlists; see Agent tool approvals.
  2. Ungrounded or invented answers under pressure — tighten retrieval and citation requirements; see Grounding and citations.
  3. Re-run the pack against the revised candidate and confirm the probe flips back to resisted — and that no previously resisted probe flipped the other way.
Because each run spends LLM tokens, the route is rate-limited; schedule full pack runs per candidate version, not per edit. To make the floor binding rather than advisory, set min_safety_score (0-100) in POST /agents/{id}/red-team/gate-settings — promotion then refuses any candidate that scores below the floor or newly compromises a probe the pinned baseline resisted.

​
4. Soak tests

A soak test drives synthetic sessions through the candidate along a staged concurrency ramp and reports what sustained load does to the conversation experience. Each session replays your script turn by turn; the ramp stacks stages so you can watch behavior degrade (or hold) as pressure rises:
Limits: at most 8 scripted turns per session, 6 stages, 50 sessions in total, and 10 concurrent sessions per stage. A full ramp therefore drives hundreds of scripted turns through the candidate — read it as one long conversation’s worth of wear, not a one-shot benchmark. That sustained repetition is exactly what surfaces the failure classes a single dry-run never shows: context and memory handling degrading in later turns, cost per conversation drifting as the agent gets chattier under pressure, and latency stretching once concurrency competes for tool calls. The report rolls the run up per stage — per-turn p50/p95/p99/max latency, cost per conversation, tool-timeout rate, containment share, optional barge-in degradation against your threshold, and the stage-over-stage degradation curve: Decide from the curve, not the averages. A candidate whose p95 doubles and whose tool-timeout rate climbs between stage 2 and stage 3 will feel fine in the practice studio and fall over during a campaign spike; hold it and fix the slow tool path or the prompt bloat first. Compare the same ramp against the baseline version and treat a sustained step-change in p95 or cost per conversation as a regression, the same way you would a failing regression test.

​
5. Gate discipline: run all three before every promotion

None of the three is worth much as a manual ritual. Wire them to the promotion path so a candidate cannot go live ungated:
  1. Configure the pre-promotion regression gate with POST /agents/{id}/promotion-gate/settings: pin the golden-set ids that must replay, set p95_regression_threshold (for example 0.2, meaning up to 20% slower is tolerated) and judge_score_regression_threshold (for example 0.05), and optionally an absolute minimum pass rate the candidate must clear.
  2. Set the red-team floor with POST /agents/{id}/red-team/gate-settings (min_safety_score).
  3. Keep a soak ramp in your release checklist and run it against every candidate that changes tools, models, or prompt structure.
Once enabled, every POST /agents/{id}/versions/{vid}/promote and every prompt rollback first replays the pinned suite and the red-team gate, and refuses with a 422 plus the gate report on a regression beyond tolerance. That is the enforcement point: the gate report in the 422 tells you which test or probe regressed, so the rollback decision is a read, not a guess. For the wider rollout shape this gate sits in, see AI agent rollout pipeline; when a promotion is refused, Troubleshooting agent promotion gate blocks walks the report fields.

​
6. Alerting and thresholds

The soak report is also your production baseline. After each promotion, keep the candidate’s stage-1 numbers — p95 latency, cost per conversation, tool-timeout rate, containment share — and compare the next candidate’s ramp against them, not against zero. Thresholds that work in practice:
  • p95 latency: alert when a candidate exceeds the baseline by more than your p95_regression_threshold; the promotion gate already enforces this for pinned suites, so mirror it for soak.
  • Cost per conversation: alert on a step-change above a few percent; cost drift is usually prompt bloat or an extra tool round-trip.
  • Tool-timeout rate: alert above 1%; timeouts are the fastest way a contained conversation becomes a handoff.
When a live agent starts drifting after promotion rather than before it, the operational runbook is Agent fleet debugging playbook — pull the execution trace for a drifting conversation, promote the failing turns into your regression corpus, and close the loop by making the next candidate pass the test that caught its predecessor.