Skip to main content

​
Triage the agent fleet with debug-log snapshots

Fleet dashboards answer “which agent is underperforming.” A support triage ticket answers “what did one conversation do wrong.” This playbook is the third thing: a runbook that links the six anomalies that recur in production to the surfaces that diagnose them — the per-turn debug log, the turn drill-in, the governance audit chain, and the Insights export — so a live-floor operator can go from alert to cause to decision without grepping logs.

​
1. Why audit by conversation, not by value

Aggregate traffic and aggregate KPI each describe a population. A support triage payload describes what the agent’s last machine did — one conversation, one turn, one tool call. Triage is where you decide whether a fleet-level gap is a real failure mode or a volume artifact. The fleet-level lens stays the agent comparison dashboard: percentile-ranked cost, containment, resolution, escalation, and margin across every agent with traffic. It picks out the outlier. Triaging the outlier at conversation level is what this playbook covers; if the outlier holds up on a cold conversation audit, you have a promotion-or-downgrade decision (section 5), not a customer-confidence problem.

​
2. Wire the per-turn snapshot

Two endpoints carry the operational debug log (30-day retention, one row per turn):
  • GET /api/v1/agents/conversations/:conversationId/debug-log — the paged list.
  • GET /api/v1/agents/conversations/:conversationId/debug-log/:turnId — one turn with its full tool-call and retrieval payloads (the list intentionally returns counts only).
Both ride the owner/admin/developer role gate, and both return 404 for a conversation id that doesn’t exist in your tenant — existence isn’t leaked through the response body. The dashboard’s per-conversation debug page composes the same endpoints; a script or runbook runner can wire them directly:
The list endpoint is keyset-paginated on turn_index: limit clamps at 500 per page, and the response emits an opaque cursor you pass back as ?cursor= until the page empties. A 30-day TTL covers the operational window; for anything longer, or when a snapshot must survive a promotion review, chain into the governance export:
That endpoint returns per-turn governance rows — the immutable ai-turn-audit chain behind the conversation’s chain-of-decisions panel — with verbatim prompts, per-turn outcome tokens (ok / escalated / refused / error / cost_capped), pointer references to the tool calls and retrieval hits, and the model + prompt/completion/cache token counts. Where a section below says “pull the audit chain,” it means this endpoint.

​
3. The turn card — read a turn by machine

A debug-log row reads the same way whether you expand it in the dashboard or pull it over the API. Order the fields as a machine card and the anomaly shapes itself: Build the card by hand in the drill-in, or assemble it from the audit row when you chain; either way the model field resolves against the provider tier — see how the curated provider lineup exports model ids in Set your organization’s LLM provider preference — and the cost figure stays the triage-level read.

​
4. Six anomaly patterns, tied to their surfaces

Each pattern below names the anomaly, how it shows up in the turn card, and the surface that closes the loop. Work the playbook in order: snapshot → isolate → remediate upstream.

​
OR-402 authgate loop

Shape: the same tool name recurs on consecutive turns with status: "error" (audit chain) or an identical-shape input and a failing output (drill-in), while the conversation keeps replying instead of escalating. A runbook runner that polls the debug-log list flags the pattern when one tool call crosses a 401-class error and the agent re-issues the call unmodified. Read: expand the turn. The tool’s credentials are missing, expired, or under-scoped. The agent can’t see the 401 — it sees a generic failure — so it retries, and the dead loop persists turn after turn until the conversation abandons. Close: the play is to escalate, not to retry. Pair this pattern with the loop-check gate below (dead-loop turns); an authorization failure is a loop with a stable cause, so the loop detector catches the bulk of them — the authgate detector narrows it. Either way, check your tool’s auth configuration, not the prompt.

​
Dead-loop turns

Shape: the loop-check chain marks consecutive turns as a semantic repeat — the customer and the agent re-asking each other the same question with no forward progress. Read: the turn_index sequence carries the same user intent with no tool dispatch and no new retrieval. The loop detector configuration page sets the repeat threshold and the action on exceed (escalate / tag / abort). Close: label the failure and gate promotions on loop-free behavior. The loop detector’s endpoint is the signal; the debug-log chain is the evidence that the loop was real and not a sparse-KB phrasing quirk.

​
Handback breach (voice)

Shape: an escalated call returned to the AI agent, and the resumption produced a wrong or incoherent first turn. The handback path is documented in Return an escalated call to the AI agent (handback). Read: the first post-handback turn carries a user_message assembled from the human agent’s wrap-up. A breach reads as retrieval on the wrong document, a tool dispatched against stale context, or an assistant response that skips what the human already resolved. Close: treat any escalation as potential handback. The audit chain’s outcome: "escalated" markers scope the conversations to check; the first-turn drill-in tells you whether the handoff payload carried the right context.

​
Guardrail concede

Shape: the grade concedes — a QA score collapse, a refused outcome where ok was expected, or an escalation into a guardrail policy that should have held. The concession is the score going soft under policy pressure, not the guardrail firing. Read: the audit row’s outcome: "refused" plus a prompt that should have been answered, or a confidence figure that collapses between consecutive turns while the policy flags stay green. Close: the fix lives in the guardrail policy or the prompt, not the model. A prompt that’s drifting into a guardrail’s scope concedes even on a premium model — re-scope the tool or relax the policy.

​
Under-elicited knowledge

Shape: the turn carries a retrieval_count of zero, or retrieval hits with low scores, while the agent answers anyway — the grounding missed and the model improvised. Read: expand the turn. The retrieval set is empty or generic; the assistant response is fluent but ungrounded, and the audit row’s retrieval_refs list is empty (no document pointer). Close: the knowledge-gap miner loop turns the missed question into a reviewed, published KB article. Triage marks the gap; the miner closes it. Gate the fix on the next debug-log pass actually retrieving the new chunk.

​
Tool-approval streaks

Shape: tool calls queue on status: "blocked" (audit chain) or sit in pending-approval state while the conversation stalls. The agent isn’t looping — it’s locked behind an approval queue no operator is draining. Read: the tool_call_refs show consecutive dispatches returning blocked, and the conversation’s latency balloons without any model-side cost. Approve or deny with a reason through the pending tool approvals queue. Close: if the same tool recurring on the queue is routine, tune the tool’s risk marking — every dispatch doesn’t need sign-off. A streak means the gate is set too tight for the tool’s actual blast radius.

​
5. Triage runbook: promote, downgrade, or fix

Decide on a snapshot, then route the fix:
  1. Block a promotion. Any of the six anomalies recurring on the canary blocks it. The rollout pipeline gates on regression checks; a failing debug-log snapshot is the evidence a stage shouldn’t advance.
  2. Downgrade to a cheaper model. A turn-price card (section 3) is the evidence. If the premium model is conceding to guardrails or improvising on an empty retrieval set, the deficit is prompt/knowledge, not model class — a cheaper preset on the model presets page will perform the same at lower cost while the knowledge fix lands. If grade holds on the cheap preset across a snapshot window, the card says the downgrade is safe.
  3. Fix upstream. Under-elicited knowledge → the knowledge-gap loop. Authgate → tool credentials. Approval streak → risk marking. Loop detection → loop detector configuration. Handback breach → handoff payload.
Route by the cheapest correct surface. Don’t tune the prompt when the retrieval set is empty, and don’t upgrade the model when the tool gate is starving the queue.

​
6. Writing the report

Triage findings ship as a snapshot, not as a notebook. Mirror the agent comparison endpoint pattern — serialize the exact comparison the operator was reading, so the report is reproducible:
The report names the agent, the anomaly class from section 4, the evidence (the debug-log conversation and turn id, or the audit row id), the resolution route from section 5, and the snapshot above so the before/after ranking is recorded. When the same agent re-earns a promotion review, the report is the chain of evidence — a blocked canary needs a clean turn-price card on re-run, not a narrative.