> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Tune AI-Agent Guardrails with Analytics

> Work the guardrail analytics and citations-audit surfaces as a loop — connect experience to allowed-vs-blocked outcomes, tune one rule at a time, set tone per check rather than per agent, and export incidents to the audit log — without disabling the whole policy.

# Tune AI-Agent Guardrails with Analytics

The [AI agent rollout pipeline](/guides/ai-agent-rollout-pipeline) attaches
a guardrail policy to your agent before launch, and the
[guardrail effectiveness](/agents/guardrail-effectiveness) reference
explains the monitoring surface's fields. This guide is the missing tuning
loop: how to read the dashboards in production, adjust one rule at a time,
set the tone of a refusal per check rather than per agent, and leave an
audit trail your compliance review can replay — without stripping coverage
from the policy.

The endpoint paths below are relative. Send them against
`https://api.orbit.devotel.io/api/v1`.

## 1. What the Live page shows

Open **Dashboard → AI Agents → Guardrail Effectiveness**
(`/agents/guardrail-analytics`). The page visualises the same
`GET /agents/guardrail-analytics` rollup described in the
[guardrail effectiveness reference](/agents/guardrail-effectiveness). Read it as
a live triage list: per turn, per hourly sweep, which rule fired on which
conversation. When a support ticket lands as "the agent refused my refund
question," this page resolves it — you see the firing as it happened, by
rule, ranked by how badly the policy mis-fired this hour.

Read the page in this order:

1. **Recent firings** — conversations a guardrail stopped in the current
   window, newest first, with the rule id and the outcome (`refused`,
   `cost_capped`).
2. **Totals** — fleet-wide firing counts and rates.
3. **By policy** — which applied policy owns the firing.
4. **Trend** — whether the rate is rising or flat.

Pair it with the citations side: the **Citations audit** tab on the agent's
page grades each turn `ok` / `no_marker` / `orphan_marker` /
`empty_retrieval`. Guardrails tell you what the agent *refused*; citations
tell you what it *cited*. Run one surface for the policy, the other for
grounding, and you cover both halves of "is the agent behaving."

## 2. Connect user-allowed vs blocked rules

Every rollup row names the policy and, where your config permits, the rule
id of the check that fired. Tune against the experience each rule produced:

* **Blocked** — the turn was stopped and the agent returned a refusal.
* **User-allowed** — the turn completed normally in the same traffic.

Two modes fire in different windows, and the analytics surface reports each
against the same conversation count — a firing that is `blocked` (hard
stop) can sit beside an `allowed` firing (soft flag) on the same policy.
Read the rate as the decision you made per rule: `allowed` means you are
watching without stopping; `blocked` means you are stopping. Treat
`allowed`-mode firings as measurements, not failures.

Pull the rollup and walk `by_policy` and `by_agent` down to the rule id:

```bash theme={null}
curl -s "https://api.orbit.devotel.io/api/v1/agents/guardrail-analytics?agentId=ag_123&days=14" \
  -H "Authorization: Bearer $ORBIT_API_KEY"
```

A firing with no policy stamp (the null bucket in `by_policy`) — an agent
outside the matrix, or a raw compliance/safety check with no coverage row —
means fix the coverage before you tune the rule.

## 3. Tune one rule, not the policy

The rollout guide's warning holds in production: **never disable the whole
policy**. The endpoints are the same ones the pipeline attached — the
analytics surface reads them back; the tuning write runs against the policy
library, not the agent's whole config.

For each rule id the rollup surfaces, pick one move:

| Outcome                            | Tune                          | Fall-back                       |
| ---------------------------------- | ----------------------------- | ------------------------------- |
| `refused` too often (false blocks) | Loosen the rule matcher       | Soft-scope the refusal copy     |
| `refused` too rarely (leaks)       | Tighten the rule matcher      | Escalate to a human queue       |
| Clean `allowed` firing             | Promote to blocked            | The policy's standard fall-back |
| Bad `allowed` firing               | Keep allowed as a metric only | Log-only, no fall-back change   |

Apply the change and re-measure the next window; the trend bucket resolves
the edit into a changed rate by the next day.

### Set tone per check, not per agent

Guardrail tone is attached to the individual rule, not to the agent's whole
config. One agent running a policy with several checks — a refund-scoping
rule, a profanity rule, a prompt-injection rule — should refuse very
differently per category. The "AI analyst" voice — firm-and-brief, or
polite-scope, or deflect-to-human — belongs on the rule id, so two agents
sharing one policy sound consistent on the same rule while an agent with no
policy falls back to a generic refusal.

Keep a small mapping table current with the policy:

| Rule               | Tone             | Fall-back                                                  |
| ------------------ | ---------------- | ---------------------------------------------------------- |
| `refund-scope`     | polite-scope     | "I can only discuss refunds for purchases on this account" |
| `profanity`        | firm-brief       | hard refusal, no apology                                   |
| `prompt-injection` | neutral-redirect | deflect to a human queue                                   |
| `sensitive-topic`  | soft-defer       | offer a hand-back to a live agent                          |

The whole-policy disable button is the synonym for removing coverage from
every agent the policy guards; narrowing a rule keeps the coverage while
you fix the shape it caught wrongly. Tune the rule, not the policy.

## 4. A worked loop

1. The Live page shows seven refusals on `refund-scope` this hour.
2. Pull `GET /agents/guardrail-analytics?agentId=ag_123&days=1` to confirm
   the spike and rank the rule.
3. Compare the turns against the rule's fall-back copy — the agent refused
   to discuss refunds *at all* on messages citing a refund amount.
4. Edit the `refund-scope` rule: require the amount in a claim position,
   not in quoted context. Set its tone to the `polite-scope` row above.
5. Check `by_policy` the next day — the trend line drops back under the
   workspace's baseline. Coverage on every other rule in the policy never
   moved.

## 5. Export incidents to the audit log

Every write the policy surface makes — applying a policy, editing a rule,
toggling a check, changing a fall-back — lands in the workspace's audit log
with the actor and timestamp, so the firing you saw on the Live page and
the fix you applied are attributably paired. Filter the audit log by the
policy and agent ids to reconstruct who tuned which rule when. Longer-form
exports (a segment or conversation dump) write their own audit entries, so
the export a compliance review pulls is attributable back to the person who
pulled it. See the [audit log guide](/guides/audit-log) for filtering and
export mechanics.

Guardrail *firings* themselves do not each land as audit rows — a firing is
a per-turn runtime event, not a management action — so the audit log records
the *controls* that shaped the firing, while the firing itself lives on the
turn record the runtime writes. That pairing keeps a compliance review one
short hop per incident.

## Related reading

* [AI agent rollout pipeline](/guides/ai-agent-rollout-pipeline) — attach
  the policy and run the pipeline before launch.
* [Guardrail effectiveness reference](/agents/guardrail-effectiveness) —
  the rollup endpoint's fields and metric semantics.
* [Grounding and citations](/agents/grounding-citations) — the citations
  audit side of the live loop.
* [Audit log](/guides/audit-log) — where rule edits and exports land.