> ## Documentation Index > Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt > Use this file to discover all available pages before exploring further. # Guardrail effectiveness monitoring: read firing rates and tune policies > Monitor whether your AI agents' guardrails are catching violations on live traffic — refusal and block rates, per-policy effectiveness, and violation trends — then tune the policy matrix from what you learn. # Guardrail effectiveness monitoring Enabling a guardrail policy is only half the loop. A policy that fires on legitimate traffic refuses customers it should serve; a policy that never fires is a gap in your safety matrix. The **Guardrail Effectiveness** surface closes that loop: it rolls up every completed agent turn in a time window and shows you, per agent and per policy, how often guardrails fired and what kind of firing it was. The endpoint paths below are relative. Send them against `https://api.orbit.devotel.io/api/v1`. ## Why monitoring matters Two failure modes sit at opposite ends of the firing-rate spectrum, and both hurt: * **Over-firing.** A policy with a high refusal rate is turning away real customers. This shows up in support tickets as "the agent refuses to answer" long before it shows up anywhere else — unless you watch the rate. * **Under-firing.** A policy that fires at 0% against traffic you *know* attempts the violation is a hole. Rough phrasings and prompt-injection probes pass through as normal turns. Without the rollup, both modes are invisible: turns complete, conversations end, and nobody aggregates the outcomes. Read the firing rates regularly the same way you read delivery rates on messaging — the config surface tells you what *can* fire; this surface tells you what *did* fire. ## What the rollup returns `GET /agents/guardrail-analytics` aggregates every completed agent turn in the window into five blocks: | Block | Contents | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `window` | The resolved time range (`from` / `to`) and the low-confidence threshold used for grounding stats. | | `totals` | Fleet-wide outcome counts (`ok`, `refused`, `escalated`, `error`, `cost_capped`), the derived firing rates, and grounding-confidence stats. | | `by_agent` | One row per agent with its applied policy, outcome counts, rates, and grounding stats. Sorted by refusal count, capped at `agentLimit` (default 100, max 500). | | `by_policy` | One row per guardrail policy with the number of distinct agents it governs, aggregate counts, and rates. Agents running with no policy stamp group under a null bucket — an explicit "no coverage" row. | | `trend` | Daily buckets (`YYYY-MM-DD`) of each firing count, so refusal and low-grounding trends over time are visible. | Every rate is a decimal in `[0, 1]` rounded to four places. An empty window returns zero rates rather than errors. ## Where it lives **Dashboard → AI Agents → Guardrail Effectiveness** (`/agents/guardrail-analytics`). The page is a thin visualization of this same endpoint — operators read the charts; anything you build reads the API directly. Reading it requires an **owner or admin** role with the `agents:read` scope. The rollup surfaces the whole workspace's agent decision behavior, so it is gated the same way as the tenant-wide turn-audit export. The endpoint is read-only, rate-limited to 30 requests per minute, and writes nothing. ### Query parameters | Param | Type | Default | Notes | | ------------------------ | ------------- | ---------------- | ---------------------------------------------------------------------------------------- | | `agentId` | string | — | Narrow the rollup to a single agent. | | `from` | ISO-8601 | — | Inclusive lower bound. When omitted, the window is `days` long, ending at `to` (or now). | | `to` | ISO-8601 | now | Exclusive upper bound. | | `days` | integer 1–365 | 30 | Window size when `from` is not supplied. | | `lowConfidenceThreshold` | number 0–1 | platform default | Grounding-confidence cutoff for the low-confidence counters. | | `agentLimit` | integer 1–500 | 100 | Top-N per-agent rows, ranked by refusal count. | Example — the last two weeks for one agent: ```bash theme={null} curl -s "https://api.orbit.devotel.io/api/v1/agents/guardrail-analytics?agentId=ag_123&days=14" \ -H "Authorization: Bearer $ORBIT_API_KEY" ``` ## The tuning loop Treat guardrail tuning as a loop, not a launch checklist: 1. **Read the rollup.** Start at the totals to see fleet-wide firing, then scan `by_policy` for outliers — a policy whose refusal rate towers over the rest, or a row sitting at 0% on heavy traffic. 2. **Drill into the agent.** Pass `agentId` to isolate one agent's rates and check whether the outlier policy is one bad agent pairing or a fleet-wide problem. 3. **Check the trend.** Is the firing rate rising (a new attack pattern, or a drifted prompt) or flat (a policy mis-set since day one)? 4. **Fix the policy.** High false-block rate → rewrite the policy's rules or drop it from the agent. Low firing rate on traffic it should catch → strengthen the policy or confirm it is actually applied to the agent (agents with no policy stamp appear in the null bucket). 5. **Re-measure.** The effect of your change shows up in the next day's trend bucket. Do not tune blind twice. ## Wiring it to the config surface The dashboard splits the loop across two surfaces: * **Per-agent page → guardrail config** — the policy library and per-agent safety toggles: which policy applies to this agent, and which categories are on. * **AI Agents → Guardrail Effectiveness** — this monitoring surface: whether the applied policies are firing and at what rate. Work them against each other. The `applied_policy_id` / `applied_policy_name` columns in `by_agent` connect the two: when a row shows a bad firing rate, open that agent's config, adjust the policy or toggles, and come back to the monitoring page after the next day's traffic. ## Metric semantics The outcome vocabulary is small and each term maps to observable agent behavior: * **`ok`** — the turn completed normally, no firing. This is the residual outcome; per-turn non-firing checks don't each count separately. * **`refused`** — a guardrail (policy, compliance, or safety check) stopped the turn. The agent returns a refusal-shaped reply instead of serving the request: the user gets a soft "I can't help with that" rather than an error or silence. This is the primary firing signal. * **`cost_capped`** — a cost ceiling stopped the turn (per-run or per-conversation cap). The turn hard-stops at the limit. * **`escalated`** — the turn was handed to a human per its escalation rules — neither served by the agent nor refused by a guardrail. * **`error`** — the turn failed on an exception, not a guardrail. Two rates are derived and worth being precise about: | Rate | Definition | | -------------- | -------------------------------------------------------------------------------------------- | | `refusal_rate` | `refused / total` — soft-stop firings. | | `block_rate` | `(refused + cost_capped) / total` — turns the guardrail layer hard-stopped by any mechanism. | The **trend window** defaults to the last 30 days, bounded to a maximum of 365, and each trend point is a UTC day bucket. Rates are computed from raw counts against that window — compare two windows, never two raw counts. The grounding block is separate from firing: it reports how confident the agent's factual grounding was on the turns that measured it, and how many of those turns fell below your `lowConfidenceThreshold`. A high `low_confidence_rate` with a low `refusal_rate` is the classic "answers confidently, wrongly" shape — tune the grounding policy up, not the refusal policy down. ## Worked example A support workspace reads its 30-day totals: ```json theme={null} { "by_policy": [ { "policy_id": "gp_prompt_injection", "policy_name": "Prompt injection", "agents": 12, "counts": { "total": 41000, "ok": 40182, "refused": 800, "escalated": 0, "error": 3, "cost_capped": 15 }, "rates": { "refusal_rate": 0.0195, "block_rate": 0.0199 } }, { "policy_id": "gp_profanity", "policy_name": "Profanity", "agents": 12, "counts": { "total": 41000, "ok": 40987, "refused": 10, "escalated": 0, "error": 3, "cost_capped": 0 }, "rates": { "refusal_rate": 0.0002, "block_rate": 0.0002 } } ] } ``` Two signals: 1. **Prompt injection fires at \~2%** on otherwise-clean support traffic. Real customers are being soft-refused \~820 times. The trend shows day-14/15 spikes — a campaign reply contained the phrase "ignore previous instructions" in customer quotes, and the policy misfired on it. Action: tighten the policy's block rule to require injection markers in *instruction position* (leading the message), not in quoted context, and re-measure the next window. 2. **Profanity fires at 0.02%** on traffic known to include abusive callers. Action: check the null bucket in `by_policy` — it turns out two agents carry no policy stamp at all. Apply the policy to those agents; the profanity rate was low because two of twelve agents were outside the matrix, not because the traffic is clean. One read of the rollup found both an over-firing and an under-firing policy. That is the entire point of the surface. ## Related reading * [Agent cost controls](/agents/cost-controls) — the toggles side of the matrix: per-run and per-conversation cost caps and tool-loop limits, whose firings show up here as `cost_capped`. * [Continuous production-call eval sampling](/agents/continuous-production-evals) — replay a deterministic slice of production calls against golden sets, the quality counterpart to guardrail firing rates. * [Grounding and citations](/agents/grounding-citations) — what the grounding-confidence stats in this rollup measure.