> ## Documentation Index > Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt > Use this file to discover all available pages before exploring further. # Runtime turn classifiers and eval rubrics: how agent turns are judged > The runtime safety family behind live AI agents — a second-pass prompt-injection check on the input, a scope-drift check on the output, a frustration-attribution layer over the existing sentiment signals, their shared fail-open contract, and the tool_selection rubric that grades an agent's tool choices in LLM-judged evals. # Runtime turn classifiers and eval rubrics: how agent turns are judged [Guardrail policies and red-team gates](https://docs.orbit.devotel.io/concepts/guardrails-and-red-team-pipeline.md) govern an agent before it ships: the policy library, the token budgets, the red-team pack, and the promote-time gates. Once the agent is live, a second family of checks watches individual turns. Three runtime classifiers look at each turn — one guards what comes in, one guards what goes out, and one explains *who* an unhappy customer is actually unhappy with — and on the eval side, the `tool_selection` rubric grades whether the agent reached for the right tools. This page explains what each of them does, the fail-open contract they all share, and how to read their results. ## 1. Input vs output guards The two content classifiers are mirror images of each other: the injection second-pass guards the **input** against a hostile caller, and the scope-drift analyzer guards the **output** against an agent that has wandered. ### The injection second-pass guards the input Every agent input already walks a static pattern layer — regex heuristics plus a high-entropy detector that catches base64, hex, and random-looking smuggling payloads. That layer is the hard guarantee and is always on. The LLM second-pass is the smarter pickup on top of it. It is opt-in behind the `DEVOTEL_PROMPT_INJECTION_LLM_CHECK_ENABLED` environment gate (off by default) and costs one fast-model call per eligible turn. It runs when the pattern layer did not already flag the input, or when the input is long enough that a sophisticated injection could hide inside legitimate-looking text. The classifier looks for attempts to override the system instructions, extract the system prompt, assume a different persona, exfiltrate data or tool outputs, smuggle instructions through encoded content, or bypass safety rules. Very short inputs skip the second pass entirely. When the second pass fires it returns a verdict with a 0-100 confidence, a category, and a rationale. When it errors, the turn degrades to the pattern layer only — a classifier outage can never break the agent runtime. ### The scope-drift analyzer guards the output The scope-drift analyzer asks the mirror-image question: is the agent's own reply consistent with the scope its system prompt declares? It flags a reply that: * answers a topic the scope does not cover (`off_topic`), * commits to a policy, price, refund, or promise the scope never grants (`unauthorized_advice`), * abandons the declared persona or role mid-conversation (`persona_break`). Declining a question or handing off to a human is in-scope behaviour and is not flagged. The analyzer deliberately reuses the agent's **existing instructions** as the scope definition. There is no separate scope field for you to fill in and keep in sync: a second source of truth would drift from the prompt that actually governs the agent, and the analyzer would end up measuring the wrong thing. Short replies are skipped. A response under 24 characters ("Sure, one moment") can never meaningfully drift, so the analyzer returns no-finding and keeps the cost off acknowledgements. An agent with an empty scope is skipped for the same reason: there is nothing to measure against. ## 2. Frustration attribution as a layer The existing signals already detect *that* a customer is unhappy: the sentiment scorer and the loop detector's proxies (re-asking, a sentiment crater, "talk to a human" requests). What they cannot tell is **who** the frustration is aimed at, and that distinction decides the right action: * "This bot is useless, you're not listening" — the **agent** is the problem. Another AI turn makes it worse; hand off to a human. * "I've been overcharged twice, this is absurd" — the **situation** is the problem. The customer is extremely angry, but the agent may be doing fine and should keep helping. Escalating both identically is what makes handoff thresholds feel arbitrary. Tune them to catch the first case and every angry-about-billing customer gets yanked to a human; tune them the other way and the customer shouting at the bot keeps getting bot. The frustration-attribution classifier is the layer that answers that question. It reads the customer's latest message plus the recent turns and returns a target of `ai`, `situation`, or `none`. It layers on top of the `sentiment_below` trigger and the loop detector; it does not replace them. The existing triggers still decide *that* something is wrong, and the attribution tells the caller *which* of the two cases it is looking at. When unsure, the classifier prefers `situation`, because wrongly blaming the agent causes needless handoffs. ## 3. The shared fail-open contract All three classifiers share one contract: **a classifier error never blocks a healthy conversation.** These analyzers watch an agent's own output and a customer's tone on the hot path; an LLM outage must degrade to "no finding" and let the conversation continue. Concretely: * the injection second-pass degrades to pattern-only on any error, * the scope-drift analyzer returns `drift: false`, * the frustration analyzer returns `target: "none"`, leaving the pre-existing sentiment and loop triggers to behave exactly as they do today. This is the opposite posture of the promote-time red-team gate, which is fail-closed on purpose: a safety gate that fails open is worse than none, because a crashed gate would wave an unsafe version through. A runtime analyzer that failed closed would wedge healthy conversations whenever the classifier blips — a strictly worse outcome than missing one finding. Every verdict, including the no-finding ones, is logged at info level with a structured metric — `agent.injection_classifier.verdict`, `agent.scope_drift.verdict`, and `agent.ai_frustration.verdict` — carrying the verdict, the clamped confidence, the category or target, and the tenant and agent ids. Logging the negatives matters: a rollout cannot pick a real confidence threshold from only the positives. The raw customer text and agent reply are deliberately never logged, because they may carry PII. The verdicts land in structured logs and metrics for operator pull, not enforcement. Today the classifiers advise; the enforcing guarantees remain the static pattern layer, the blocked-topic and denylist configuration, and the pre-deploy gates. That separation is what makes the staged rollout possible: operators pull per-tenant verdicts from the logs, measure confidence distributions on real traffic, and only then decide what a blocking threshold should be. ## 4. The tool\_selection eval rubric The [LLM-judged eval runner](https://docs.orbit.devotel.io/concepts/llm-judged-evaluation-ledger.md) runs an agent over a dataset and scores each row with a judge model. For every row it invokes the agent, captures the actual output, hands both to the judge under the selected rubric, and marks the row passed when the score is at or above the threshold (default **70**). Rows run at a bounded concurrency of five, and a row that errors — the agent failed, or the judge returned unparseable output — is recorded with an error message and excluded from the average score rather than scored as zero. A partial run is better than no run. The built-in rubric enum is `correctness`, `helpfulness`, `safety`, `groundedness`, `tool_selection`, and `custom`. `tool_selection` is the newest member and covers a gap the other rubrics cannot see: whether the tools the agent actually reached for were the right ones, given what the customer asked and what was available. The judge sees the tools the agent could choose from and the calls it actually made, in order, with arguments clipped so a full payload (and any PII inside it) never reaches the judge prompt. It then scores 0-100: * 90-100: every call was warranted and sufficient — the right tools, no needless ones, nothing obviously missing. * 70-89: broadly right, with a redundant call or a slightly sub-optimal choice. * 40-69: a clearly needed tool was never called, or a called tool did not fit the request. * 0-39: wrong tools for the task, or a destructive action the request never asked for. Two clauses in the rubric are load-bearing. First, **calling no tools can be perfectly correct**: when the question is answerable directly or no available tool fits, the judge scores that high, because the most common real failure is an agent that fires a lookup it did not need, and a judge that rewarded tool use would score that defect *higher* than a correct answer from context. Second, the judge grades only the **choice** of tools, not whether a tool returned good data — a correct call that failed upstream is still a correct choice. ### Reading tool\_selection against the pre-deploy harnesses [Pick a testing harness](https://docs.orbit.devotel.io/concepts/agent-testing-harness-picker.md) lists the six pre-deploy surfaces, and two of them already touch tool behaviour: dry-run and batch-simulation accept deterministic assertions such as `expected_tool_calls`. Those assertions grade only a scenario that **declared** its expected tools in advance, so they cannot run on live or unscripted traffic at all. `tool_selection` covers exactly that gap: an LLM judge decides whether the tool choices were appropriate when nobody declared the answer in advance. Persona-simulation's judge grades the conversation against your rubric criteria; `tool_selection` grades the agent's tool discipline specifically. Use the deterministic assertions to pin the flows you already know, and use `tool_selection` over a dataset of real or scripted inputs to catch the regressions nobody thought to declare — an agent that suddenly looks up everything, or silently stops calling the refund tool after a prompt change. The per-row reasoning names any tool wrongly called or wrongly skipped, so a drop in pass rate is debuggable row by row rather than a bare average. ## 5. Limits * The classifiers advise. Each is one fast-model call per turn and can be wrong in both directions; treat a single verdict as a signal, and tune against the logged confidence distribution per tenant rather than acting on one row. * Verdicts are operator observability today. This page documents only the structured-log contract; where a verdict surfaces in a dashboard, the dashboard's own page is the source of truth. * The eval runner spends real tokens: one agent run plus one judge call per row, at up to five concurrent rows. Size your dataset, and use the run cost cap, before a large batch. * Nothing on this page dispatches outbound voice or SMS. The classifiers observe turns, and the eval runner runs in the same sandbox discipline as the other harnesses. ## See also * [Guardrail policies and red-team gates: how agents are governed](https://docs.orbit.devotel.io/concepts/guardrails-and-red-team-pipeline.md) — the pre-deploy half of this story, including the guardrail-analytics rollup of live-turn violations. * [Pick a testing harness: the decision table](https://docs.orbit.devotel.io/concepts/agent-testing-harness-picker.md) — which pre-deploy harness answers which question, and how the deterministic tool assertions relate to `tool_selection`. * [The LLM-judged evaluation ledger](https://docs.orbit.devotel.io/concepts/llm-judged-evaluation-ledger.md) — how rubric judgments are recorded per call and where a pass/fail decision goes next. * [High-friction sessions reading guide](https://docs.orbit.devotel.io/guides/high-friction-sessions-reading.md) — the session-replay friction signals that pair with frustration attribution when you triage struggle. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.