Skip to main content

​
Tune AI-Agent Guardrails with Analytics

The AI agent rollout pipeline attaches a guardrail policy to your agent before launch, and the guardrail effectiveness reference explains the monitoring surface’s fields. This guide is the missing tuning loop: how to read the dashboards in production, adjust one rule at a time, set the tone of a refusal per check rather than per agent, and leave an audit trail your compliance review can replay — without stripping coverage from the policy. The endpoint paths below are relative. Send them against https://api.orbit.devotel.io/api/v1.

​
1. What the Live page shows

Open Dashboard → AI Agents → Guardrail Effectiveness (/agents/guardrail-analytics). The page visualises the same GET /agents/guardrail-analytics rollup described in the guardrail effectiveness reference. Read it as a live triage list: per turn, per hourly sweep, which rule fired on which conversation. When a support ticket lands as “the agent refused my refund question,” this page resolves it — you see the firing as it happened, by rule, ranked by how badly the policy mis-fired this hour. Read the page in this order:
  1. Recent firings — conversations a guardrail stopped in the current window, newest first, with the rule id and the outcome (refused, cost_capped).
  2. Totals — fleet-wide firing counts and rates.
  3. By policy — which applied policy owns the firing.
  4. Trend — whether the rate is rising or flat.
Pair it with the citations side: the Citations audit tab on the agent’s page grades each turn ok / no_marker / orphan_marker / empty_retrieval. Guardrails tell you what the agent refused; citations tell you what it cited. Run one surface for the policy, the other for grounding, and you cover both halves of “is the agent behaving.”

​
2. Connect user-allowed vs blocked rules

Every rollup row names the policy and, where your config permits, the rule id of the check that fired. Tune against the experience each rule produced:
  • Blocked — the turn was stopped and the agent returned a refusal.
  • User-allowed — the turn completed normally in the same traffic.
Two modes fire in different windows, and the analytics surface reports each against the same conversation count — a firing that is blocked (hard stop) can sit beside an allowed firing (soft flag) on the same policy. Read the rate as the decision you made per rule: allowed means you are watching without stopping; blocked means you are stopping. Treat allowed-mode firings as measurements, not failures. Pull the rollup and walk by_policy and by_agent down to the rule id:
A firing with no policy stamp (the null bucket in by_policy) — an agent outside the matrix, or a raw compliance/safety check with no coverage row — means fix the coverage before you tune the rule.

​
3. Tune one rule, not the policy

The rollout guide’s warning holds in production: never disable the whole policy. The endpoints are the same ones the pipeline attached — the analytics surface reads them back; the tuning write runs against the policy library, not the agent’s whole config. For each rule id the rollup surfaces, pick one move: Apply the change and re-measure the next window; the trend bucket resolves the edit into a changed rate by the next day.

​
Set tone per check, not per agent

Guardrail tone is attached to the individual rule, not to the agent’s whole config. One agent running a policy with several checks — a refund-scoping rule, a profanity rule, a prompt-injection rule — should refuse very differently per category. The “AI analyst” voice — firm-and-brief, or polite-scope, or deflect-to-human — belongs on the rule id, so two agents sharing one policy sound consistent on the same rule while an agent with no policy falls back to a generic refusal. Keep a small mapping table current with the policy: The whole-policy disable button is the synonym for removing coverage from every agent the policy guards; narrowing a rule keeps the coverage while you fix the shape it caught wrongly. Tune the rule, not the policy.

​
4. A worked loop

  1. The Live page shows seven refusals on refund-scope this hour.
  2. Pull GET /agents/guardrail-analytics?agentId=ag_123&days=1 to confirm the spike and rank the rule.
  3. Compare the turns against the rule’s fall-back copy — the agent refused to discuss refunds at all on messages citing a refund amount.
  4. Edit the refund-scope rule: require the amount in a claim position, not in quoted context. Set its tone to the polite-scope row above.
  5. Check by_policy the next day — the trend line drops back under the workspace’s baseline. Coverage on every other rule in the policy never moved.

​
5. Export incidents to the audit log

Every write the policy surface makes — applying a policy, editing a rule, toggling a check, changing a fall-back — lands in the workspace’s audit log with the actor and timestamp, so the firing you saw on the Live page and the fix you applied are attributably paired. Filter the audit log by the policy and agent ids to reconstruct who tuned which rule when. Longer-form exports (a segment or conversation dump) write their own audit entries, so the export a compliance review pulls is attributable back to the person who pulled it. See the audit log guide for filtering and export mechanics. Guardrail firings themselves do not each land as audit rows — a firing is a per-turn runtime event, not a management action — so the audit log records the controls that shaped the firing, while the firing itself lives on the turn record the runtime writes. That pairing keeps a compliance review one short hop per incident.