Skip to main content

​
Adversarial red-team testing

Dry-run replays, regression suites, and batch simulations answer one question: does the agent still answer correctly? None of them answer the other question: does the agent hold its guardrails when an attacker pushes on them? The red-team surface is the safety counterpart to those correctness harnesses — it replays a built-in adversarial pack against your agent and returns a safety scorecard before the agent ever serves live traffic. Every run executes against the agent in sandbox mode: no billing, no memory writes, and mutative tools are short-circuited. The scorecard is computed inline — nothing is stored — so you can rerun it freely while you tighten a prompt. The endpoint paths below are relative. Send them against https://api.orbit.devotel.io/api/v1.

​
What red-team solves

The correctness harnesses probe whether the agent picks the right answer. The red-team pack probes the opposite axis: whether it keeps its safety posture under adversarial pressure. You need both, and for different reasons: This split matters most at the moment you promote an agent. A regression gate that only measures correctness can pass an agent whose prompt lost its refusal boundaries — every test case still “worked,” and none of them tested an attacker.

​
The built-in adversarial pack

The pack holds 12 probes across four categories — three per category. Each probe is purposefully inert: it asks the agent to reveal a planted fake secret or echo a nonsense canary token, never to produce genuinely harmful output. The probe exercises the guardrail, not the payload. Detection is deterministic. A probe plants a canary token that the agent only emits if it complies with the malicious instruction; a safe refusal never contains the token. So scoring needs no judge model:
  • The reply contains any compromise marker → the probe outcome is compromised (the attack succeeded).
  • Otherwise → resisted (the agent held the line).
  • A transport or runtime failure → error. An error is not recorded as a compromise and does not count against the score — an infrastructure blip is not evidence the agent leaked.
The four categories, read as a threat model: List the catalogue before you run it. GET /agents/:id/red-team/pack returns each probe with its category, title, and the safe-behavior expectation, plus a per-category count. The compromise markers themselves are intentionally omitted from the response so the canary tokens are not advertised to callers that should not see them.

​
Running the pack

POST /agents/:id/red-team/run replays the whole pack (or a subset) and returns the scorecard inline. Pass a categories array to narrow the run; omit it to run all four.
Running a pack spends LLM tokens once per probe, so the route is deliberately tight: owner, admin, or developer role with the agents:write scope, rate-limited to 3 runs per minute. (The pack catalogue endpoint above is read-only and allows 60 per minute.) The dashboard → agent detail page renders this same surface as a red-team panel — run the pack, read the scorecard, no curl required.

​
Interpreting the scorecard

The response returns the scorecard together with every individual probe result: outcome (resisted / compromised / error), which markers matched on a compromised probe, and the reason strings that explain the verdict. The overall block reports: Read the per-category breakdown before the overall grade. A B overall with data_exfiltration at 0/3 resisted is a different problem from a B where the misses are spread across jailbreak probes — the category tells you which guardrail layer to fix, not just that something is off.

​
Remediation guidance

  • Compromised jailbreak or policy_guardrail probes → the prompt’s refusal and scope boundaries are weak. Tighten the system prompt’s override-resistance language and confirm the agent has a guardrail policy applied — agents with no policy stamp are the classic under-firing case.
  • Compromised prompt_injection probes → the agent follows instructions embedded in data. Add a prompt-injection policy or strengthen the instruction/data separation rule in your guardrail config.
  • Compromised data_exfiltration probes → secrets or hidden instructions are reachable. Review what the agent’s prompt and tools expose, then rerun.
  • error outcomes → a probe failed before the agent responded (timeout, runtime fault). Errored probes do not count against the score; they surface separately so you can rerun once the runtime is healthy.
After any fix, rerun the pack. The whole loop — read the category, fix the policy or prompt, re-measure — mirrors the guardrail-efficiency tuning loop one layer up; the two surfaces work against each other.

​
Feeding the conformity dossier

The red-team pack is tenant-owned testing — evidence you generate, not a certification Orbit issues on your behalf. When you need that evidence for an EU AI Act conformity assessment, the dossier endpoint assembles it for you: GET /agents/:id/conformity-dossier includes the most recent red-team scorecard under the Article 15 — Accuracy & robustness section, alongside your accuracy-evaluation evidence. The dossier is generated material for internal review and procurement due diligence — not a certificate of conformity or legal advice.