Skip to main content

​
Read Guardrail Analytics per Policy

The guardrail effectiveness reference explains the monitoring surface’s fields, and the tuning guide closes the rule-edit loop. This guide covers the measurement unit both depend on: the per-policy rate. A fleet-level number is an average across policies, and the average is where a silent policy hides. The endpoint paths below are relative. Send them against https://api.orbit.devotel.io/api/v1.

​
1. Why the fleet number misleads

The fleet refusal rate weights every policy by the traffic its agents see. A policy applied to one quiet agent, or guarding a pattern nobody has sent yet, contributes almost nothing to either side of the rate. “We blocked 98% of jailbreak attempts” can therefore be true while one jailbreak policy fired zero times: two healthy policies earned the number and the third watched. Read the fleet rate as context. Read the per-policy columns as the measurement. Any report that cites an aggregate guardrail rate without the per-policy breakdown is citing the average of unknown terms.

​
2. The per-policy reading

Open Dashboard → AI Agents → Guardrail Effectiveness (/agents/guardrail-analytics) and pick the look-back window, 7 to 90 days. The per-policy breakdown reports every applied policy four ways:
  1. Refusal rate by policy — the share of turns each policy refused, across the agents carrying it.
  2. Block actions by policy — hard stops per policy: refusals plus cost-capped turns.
  3. Violation trend — daily refusal and block counts over the window, per policy.
  4. Per-agent cut — each agent’s refusal rate with the policies applied to it, so a rate is attributable to “which policy on which agent.”
The same rollup is available to scripts and exports:
Walk by_policy and by_agent down to the policy you care about. The rollup attributes firings by policy and by agent, deliberately not by individual conversation; turn-level detail stays in the conversation record, and section 4 uses that division of labor.

​
3. Three signatures worth memorizing

The dead rule. A policy at zero firings over the window, on agents that see real traffic, is a question, not an answer: absence of violations, or absence of detection? A fraud denylist should ideally never fire; a matcher authored against the wrong string also never fires. Check whether the guarded pattern appears in recent transcripts, then evaluate the rule text against a verbatim violating utterance. If the pattern is in the traffic and the matcher misses it, the zero is a detection failure. The per-policy row is the only place that zero was visible. The over-triggering rule. A refusal rate double the fleet baseline on one agent is the false-positive profile. The breakdown names the policy carrying the rate, so the fix is a rule edit, never disabling the whole policy. Before you edit, capture the misfired turns; they become regression cases in section 5. The drifting rule. A violation trend rising on one policy while its neighbours stay flat is drift, and it has two causes: adversary adaptation (probers change phrasing) or data shift (a new product line sends legitimate utterances the rule was not scoped against). The signature that separates drift from a one-day surge is that it survives a window change, the 7-day reading agreeing with the 30-day.

​
4. From a violation cluster to sample conversations

When one policy spikes on one agent, the cluster’s coordinates are the policy and agent names from the breakdown. Pull the sample conversations behind the cluster from the conversation record and read them turn by turn; the conversation debug guide covers opening and tracing those conversations. A handful of spot-checked turns converts a rate into a verdict: whether what changed is the traffic, the adversary, or the rule.

​
5. Regression-test before you edit the DSL

For an over-triggering or drifting rule, add the misfired turns as regression cases on the agent’s evals page, /agents/<agent-id>/evals, before touching the rule text. The evals datasets loop covers building the dataset, and the simulation eval suite covers running it. The cases must encode both sides: the utterance the rule must stop refusing, and the violation it must still catch. Then edit the rule in the custom guardrail DSL and publish a new version. The eval run grades the edit against the captured turns before live traffic does; the per-policy trend grades it after. If the rate returns to baseline inside the next window and the regression cases hold, the revision was the fix.

​
6. Patterns on the trend panel

Seasonal spikes. A refund-heavy week raises the refund-scope rate because more traffic is about refunds, not because probes increased. The rate rises while spot-checked refusals stay correct. Check the topic’s share of traffic before tightening anything. Model upgrade effects. Changing an agent’s base model shifts refusal and grounding rates across several policies on the same day, fleet-wide. Simultaneity across policies on the upgrade date is the signature. Re-run the eval suite against the new model before attributing the shift to a rule. The jailbreak wave. One policy spikes while its neighbours stay flat, the firings cluster in a narrow time window, and the spot-checked conversations share phrasing. Capture the phrasings as eval cases, tighten or extend the rule, publish a version, and watch the trend return to baseline with coverage still applied.