> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting: agent eval-queue failures

> Diagnose agent-eval queue failures — publishing gates (EVAL_GATE_FAILED, EVAL_REGRESSION_GATE_FAILED), dataset labeling limits (LABELING_NO_INPUT, LABELING_DATASET_FULL), run comparisons (EVAL_COMPARE_SAME_RUN, EVAL_RUNS_AGENT_MISMATCH) — and the pending-run backlog that stalls evals.

# Troubleshooting: agent eval-queue failures

Agent evaluation runs traverse a queued pipeline — you submit a run against a
golden set or dataset, it sits in `queued` until the eval worker picks it up,
moves to `running`, and settles on `completed` or `failed`. Errors surface at
three points: the **publish gate** (Agent Studio publish blocked), **dataset
labeling** (annotation write rejected), and **run comparison** (compare rejected).
When a run never moves past `queued`, that usually means the backlog is
blocked on one of the codes below.

The full list of error codes lives in the [error-reference](/reference/error-codes);
this page gives you the fix for each eval-related code.

## How the agent-eval queue works

Voice-agent evaluation has two queue sources; both land in the
`voice_eval_runs` table and move through the same lifecycle
(`queued → running → completed|failed`):

* **Manual runs** — you POST `/agents/voice-eval/runs` with a golden set or
  dataset. The run is queued; a worker drains it.
* **Continuous production sampling** — an organization-wide policy
  (`GET|PUT /agents/voice-eval/sampling-policy`) samples a fixed percent
  (`sample_percent`, 0–100) of completed production calls into eval runs
  against a configured golden set. Runs are created automatically and drain
  through the same queue.

Two throttles bound the queue drain rate:

* **Per-case concurrency** — a run executes its cases (each a golden-set prompt
  or dataset row) with a bounded parallelism of 5 at a time, so a run never
  floods the LLM-judge or STT/TTS legs. Increasing `sample_percent` increases
  backlog depth without speeding up individual runs.
* **Rubric rubric load** — a run scoring the `groundedness` rubric issues the
  LLM-judge call for each case; the judge is the slowest leg and a
  `groundedness`-heavy golden set takes longer to drain than pure
  `correctness`/`helpfulness` sets.

Get the queue's depth and the runs waiting on it with the status filter:

```bash theme={null}
curl -G https://api.orbit.devotel.io/api/v1/agents/voice-eval/runs \
  -H "X-API-Key: dv_live_sk_..." \
  --data-urlencode "status=queued"
```

The response lists every pending run, newest first — a large `items` array at
`queued` means the drain is backlogged; a failure code on the settled run
(`failed`) means one of the errors below fired mid-run.

## Decision table — eval error → fix

| Error                         | HTTP | Cause                                                                                                                                          | Fix                                                                                                                                 |
| ----------------------------- | ---- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `EVAL_GATE_FAILED`            | 422  | Agent Studio publish blocked — the agent has no completed eval run meeting the configured minimum pass rate.                                   | Run an eval against the agent (`POST /agents/voice-eval/runs`), then publish once the run completes at or above the pass threshold. |
| `EVAL_REGRESSION_GATE_FAILED` | 422  | Publish blocked — the latest eval run regressed beyond the configured tolerance versus the baseline run.                                       | Investigate the failed golden cases (`GET /agents/voice-eval/runs/:id`), fix the agent, re-run evals, then publish.                 |
| `LABELING_NO_INPUT`           | 422  | POST `/agents/:agentId/labeling/annotations` had nothing to label — the conversation has no user message and no explicit `input` was supplied. | Pass `input` in the request body, or skip this conversation. Filter out empty-only (assistant-only) conversations before labeling.  |
| `LABELING_DATASET_FULL`       | 409  | The labeling dataset reached its row ceiling.                                                                                                  | Run an eval against the dataset and export it, then archive the dataset (or start a new one) before labeling more.                  |
| `EVAL_COMPARE_SAME_RUN`       | 400  | `GET /agents/:agentId/evals/runs/compare?run_a_id=...&run_b_id=...` was passed the same run id in both slots.                                  | Pass two distinct run ids. Compare a baseline vs. a candidate.                                                                      |
| `EVAL_RUNS_AGENT_MISMATCH`    | 400  | Compare runs from different agents — one or both of `run_a_id`/`run_b_id` belong to a different agent than the one in the path.                | Compare runs of the agent named in the request path. For cross-agent comparisons, export both runs and join client-side.            |

`EVAL_COMPARE_DATASET_MISMATCH` is the sibling of the last pair — the two runs
reference different datasets. Compare-only runs executed against the same
dataset.

## Monitoring queue depth

Check two surfaces together:

1. **Backlog and stuck runs:**

```bash theme={null}
curl -G https://api.orbit.devotel.io/api/v1/agents/voice-eval/runs \
  -H "X-API-Key: dv_live_sk_..." \
  --data-urlencode "status=queued" \
  --data-urlencode "limit=100"
```

A backlog that only grows means the drain is slower than the intake. The
intake source is the sampling policy (`GET /agents/voice-eval/sampling-policy`)
and manual run submissions. Lower `sample_percent` or tighten the policy's
`agent_ids` allowlist to reduce intake.

2. **Drain blockers:** fetch a `failed` run to read why it failed:

```bash theme={null}
curl https://api.orbit.devotel.io/api/v1/agents/voice-eval/runs/vevr_abc123 \
  -H "X-API-Key: dv_live_sk_..."
```

The `cases` array on a run detail carries per-case `passed`/`failed` verdicts
and `failure_reason` — read them against the table above to decide whether to
fix the agent, the dataset, or the golden set.

## See also

* [Error reference — Agent Eval section](/reference/error-codes) — the full
  error code list this page's decision table maps.
* [Continuous production evals](/agents/continuous-production-evals) — how
  the sampling policy works end to end.
* [Agent evals: datasets → runs → triage → gate](/agents/agent-versions) —
  the eval pipeline this queue drains.
* [QA evaluations and the leaderboard](/concepts/qa-leaderboard-and-evaluations) —
  the scoring pipeline eval rubrics feed.
* [Voice Agent Quality Index (VAQI)](/concepts/voice-agent-quality-vaq) — the
  latency/turn-taking metrics a `groundedness` rubric judge may reference.