Skip to main content

​
Troubleshooting: agent eval-queue failures

Agent evaluation runs traverse a queued pipeline — you submit a run against a golden set or dataset, it sits in queued until the eval worker picks it up, moves to running, and settles on completed or failed. Errors surface at three points: the publish gate (Agent Studio publish blocked), dataset labeling (annotation write rejected), and run comparison (compare rejected). When a run never moves past queued, that usually means the backlog is blocked on one of the codes below. The full list of error codes lives in the error-reference; this page gives you the fix for each eval-related code.

​
How the agent-eval queue works

Voice-agent evaluation has two queue sources; both land in the voice_eval_runs table and move through the same lifecycle (queued → running → completed|failed):
  • Manual runs — you POST /agents/voice-eval/runs with a golden set or dataset. The run is queued; a worker drains it.
  • Continuous production sampling — an organization-wide policy (GET|PUT /agents/voice-eval/sampling-policy) samples a fixed percent (sample_percent, 0–100) of completed production calls into eval runs against a configured golden set. Runs are created automatically and drain through the same queue.
Two throttles bound the queue drain rate:
  • Per-case concurrency — a run executes its cases (each a golden-set prompt or dataset row) with a bounded parallelism of 5 at a time, so a run never floods the LLM-judge or STT/TTS legs. Increasing sample_percent increases backlog depth without speeding up individual runs.
  • Rubric rubric load — a run scoring the groundedness rubric issues the LLM-judge call for each case; the judge is the slowest leg and a groundedness-heavy golden set takes longer to drain than pure correctness/helpfulness sets.
Get the queue’s depth and the runs waiting on it with the status filter:
The response lists every pending run, newest first — a large items array at queued means the drain is backlogged; a failure code on the settled run (failed) means one of the errors below fired mid-run.

​
Decision table — eval error → fix

EVAL_COMPARE_DATASET_MISMATCH is the sibling of the last pair — the two runs reference different datasets. Compare-only runs executed against the same dataset.

​
Monitoring queue depth

Check two surfaces together:
  1. Backlog and stuck runs:
A backlog that only grows means the drain is slower than the intake. The intake source is the sampling policy (GET /agents/voice-eval/sampling-policy) and manual run submissions. Lower sample_percent or tighten the policy’s agent_ids allowlist to reduce intake.
  1. Drain blockers: fetch a failed run to read why it failed:
The cases array on a run detail carries per-case passed/failed verdicts and failure_reason — read them against the table above to decide whether to fix the agent, the dataset, or the golden set.

​
See also