Skip to main content

​
Runtime turn classifiers and eval rubrics: how agent turns are judged

Guardrail policies and red-team gates govern an agent before it ships: the policy library, the token budgets, the red-team pack, and the promote-time gates. Once the agent is live, a second family of checks watches individual turns. Three runtime classifiers look at each turn — one guards what comes in, one guards what goes out, and one explains who an unhappy customer is actually unhappy with — and on the eval side, the tool_selection rubric grades whether the agent reached for the right tools. This page explains what each of them does, the fail-open contract they all share, and how to read their results.

​
1. Input vs output guards

The two content classifiers are mirror images of each other: the injection second-pass guards the input against a hostile caller, and the scope-drift analyzer guards the output against an agent that has wandered.

​
The injection second-pass guards the input

Every agent input already walks a static pattern layer — regex heuristics plus a high-entropy detector that catches base64, hex, and random-looking smuggling payloads. That layer is the hard guarantee and is always on. The LLM second-pass is the smarter pickup on top of it. It is opt-in behind the DEVOTEL_PROMPT_INJECTION_LLM_CHECK_ENABLED environment gate (off by default) and costs one fast-model call per eligible turn. It runs when the pattern layer did not already flag the input, or when the input is long enough that a sophisticated injection could hide inside legitimate-looking text. The classifier looks for attempts to override the system instructions, extract the system prompt, assume a different persona, exfiltrate data or tool outputs, smuggle instructions through encoded content, or bypass safety rules. Very short inputs skip the second pass entirely. When the second pass fires it returns a verdict with a 0-100 confidence, a category, and a rationale. When it errors, the turn degrades to the pattern layer only — a classifier outage can never break the agent runtime.

​
The scope-drift analyzer guards the output

The scope-drift analyzer asks the mirror-image question: is the agent’s own reply consistent with the scope its system prompt declares? It flags a reply that:
  • answers a topic the scope does not cover (off_topic),
  • commits to a policy, price, refund, or promise the scope never grants (unauthorized_advice),
  • abandons the declared persona or role mid-conversation (persona_break).
Declining a question or handing off to a human is in-scope behaviour and is not flagged. The analyzer deliberately reuses the agent’s existing instructions as the scope definition. There is no separate scope field for you to fill in and keep in sync: a second source of truth would drift from the prompt that actually governs the agent, and the analyzer would end up measuring the wrong thing. Short replies are skipped. A response under 24 characters (“Sure, one moment”) can never meaningfully drift, so the analyzer returns no-finding and keeps the cost off acknowledgements. An agent with an empty scope is skipped for the same reason: there is nothing to measure against.

​
2. Frustration attribution as a layer

The existing signals already detect that a customer is unhappy: the sentiment scorer and the loop detector’s proxies (re-asking, a sentiment crater, “talk to a human” requests). What they cannot tell is who the frustration is aimed at, and that distinction decides the right action:
  • “This bot is useless, you’re not listening” — the agent is the problem. Another AI turn makes it worse; hand off to a human.
  • “I’ve been overcharged twice, this is absurd” — the situation is the problem. The customer is extremely angry, but the agent may be doing fine and should keep helping.
Escalating both identically is what makes handoff thresholds feel arbitrary. Tune them to catch the first case and every angry-about-billing customer gets yanked to a human; tune them the other way and the customer shouting at the bot keeps getting bot. The frustration-attribution classifier is the layer that answers that question. It reads the customer’s latest message plus the recent turns and returns a target of ai, situation, or none. It layers on top of the sentiment_below trigger and the loop detector; it does not replace them. The existing triggers still decide that something is wrong, and the attribution tells the caller which of the two cases it is looking at. When unsure, the classifier prefers situation, because wrongly blaming the agent causes needless handoffs.

​
3. The shared fail-open contract

All three classifiers share one contract: a classifier error never blocks a healthy conversation. These analyzers watch an agent’s own output and a customer’s tone on the hot path; an LLM outage must degrade to “no finding” and let the conversation continue. Concretely:
  • the injection second-pass degrades to pattern-only on any error,
  • the scope-drift analyzer returns drift: false,
  • the frustration analyzer returns target: "none", leaving the pre-existing sentiment and loop triggers to behave exactly as they do today.
This is the opposite posture of the promote-time red-team gate, which is fail-closed on purpose: a safety gate that fails open is worse than none, because a crashed gate would wave an unsafe version through. A runtime analyzer that failed closed would wedge healthy conversations whenever the classifier blips — a strictly worse outcome than missing one finding. Every verdict, including the no-finding ones, is logged at info level with a structured metric — agent.injection_classifier.verdict, agent.scope_drift.verdict, and agent.ai_frustration.verdict — carrying the verdict, the clamped confidence, the category or target, and the tenant and agent ids. Logging the negatives matters: a rollout cannot pick a real confidence threshold from only the positives. The raw customer text and agent reply are deliberately never logged, because they may carry PII. The verdicts land in structured logs and metrics for operator pull, not enforcement. Today the classifiers advise; the enforcing guarantees remain the static pattern layer, the blocked-topic and denylist configuration, and the pre-deploy gates. That separation is what makes the staged rollout possible: operators pull per-tenant verdicts from the logs, measure confidence distributions on real traffic, and only then decide what a blocking threshold should be.

​
4. The tool_selection eval rubric

The LLM-judged eval runner runs an agent over a dataset and scores each row with a judge model. For every row it invokes the agent, captures the actual output, hands both to the judge under the selected rubric, and marks the row passed when the score is at or above the threshold (default 70). Rows run at a bounded concurrency of five, and a row that errors — the agent failed, or the judge returned unparseable output — is recorded with an error message and excluded from the average score rather than scored as zero. A partial run is better than no run. The built-in rubric enum is correctness, helpfulness, safety, groundedness, tool_selection, and custom. tool_selection is the newest member and covers a gap the other rubrics cannot see: whether the tools the agent actually reached for were the right ones, given what the customer asked and what was available. The judge sees the tools the agent could choose from and the calls it actually made, in order, with arguments clipped so a full payload (and any PII inside it) never reaches the judge prompt. It then scores 0-100:
  • 90-100: every call was warranted and sufficient — the right tools, no needless ones, nothing obviously missing.
  • 70-89: broadly right, with a redundant call or a slightly sub-optimal choice.
  • 40-69: a clearly needed tool was never called, or a called tool did not fit the request.
  • 0-39: wrong tools for the task, or a destructive action the request never asked for.
Two clauses in the rubric are load-bearing. First, calling no tools can be perfectly correct: when the question is answerable directly or no available tool fits, the judge scores that high, because the most common real failure is an agent that fires a lookup it did not need, and a judge that rewarded tool use would score that defect higher than a correct answer from context. Second, the judge grades only the choice of tools, not whether a tool returned good data — a correct call that failed upstream is still a correct choice.

​
Reading tool_selection against the pre-deploy harnesses

Pick a testing harness lists the six pre-deploy surfaces, and two of them already touch tool behaviour: dry-run and batch-simulation accept deterministic assertions such as expected_tool_calls. Those assertions grade only a scenario that declared its expected tools in advance, so they cannot run on live or unscripted traffic at all. tool_selection covers exactly that gap: an LLM judge decides whether the tool choices were appropriate when nobody declared the answer in advance. Persona-simulation’s judge grades the conversation against your rubric criteria; tool_selection grades the agent’s tool discipline specifically. Use the deterministic assertions to pin the flows you already know, and use tool_selection over a dataset of real or scripted inputs to catch the regressions nobody thought to declare — an agent that suddenly looks up everything, or silently stops calling the refund tool after a prompt change. The per-row reasoning names any tool wrongly called or wrongly skipped, so a drop in pass rate is debuggable row by row rather than a bare average.

​
5. Limits

  • The classifiers advise. Each is one fast-model call per turn and can be wrong in both directions; treat a single verdict as a signal, and tune against the logged confidence distribution per tenant rather than acting on one row.
  • Verdicts are operator observability today. This page documents only the structured-log contract; where a verdict surfaces in a dashboard, the dashboard’s own page is the source of truth.
  • The eval runner spends real tokens: one agent run plus one judge call per row, at up to five concurrent rows. Size your dataset, and use the run cost cap, before a large batch.
  • Nothing on this page dispatches outbound voice or SMS. The classifiers observe turns, and the eval runner runs in the same sandbox discipline as the other harnesses.

​
See also