Skip to main content

​
Read a voice-eval run in the dashboard

A voice-eval run replays a golden set (a named bundle of audio prompts with expected response topics) against your agent and grades every case with an LLM judge. The run detail page is where you read that evidence: the aggregate header tells you whether the run passed overall, and each case row shows you exactly how the agent answered, how long each pipeline leg took, and — when something went wrong — why. This page covers what every block on the run detail page means. For where runs come from (manual triggers and continuous production sampling), see Continuous Production Evals.

​
1. Where the page lives

Dashboard path: Agents → your agent → Testing (Evals) → Voice eval runs → pick a run. The route is /agents/:id/evals/voice-runs/:runId. The Voice eval runs table on the Evals tab lists every run for the agent with its status and pass rate. Opening a run loads its full detail — summary header, cases table, and failure diagnostics — in one page.

​
2. Run summary header

The header card aggregates the run so you can read the verdict before expanding a single case:
  • Status — Pending, Running, Completed, or Failed.
  • The Simulated pill — shown on every run (see section 5).
  • Golden set and time — the golden set id and the run’s start (or creation) time, for pinning regression deltas against the right baseline.
  • Pass rate — X/Y = Z%: passed cases over total cases. A case passes when its LLM-judge score clears the golden set’s threshold.
  • P95 latency — the 95th percentile end-to-end case latency.
  • Cost — the run’s total spend.
  • Cases — total golden-set cases executed.
  • Providers — the STT provider, TTS provider, and LLM model the run executed with, so latency and score comparisons stay apples-to-apples across candidate changes.

​
3. Per-case rows

Below the header, the Cases table holds one row per golden case:
  • Case — the case label (expanded in the drawer).
  • Actual response — the agent’s full reply text.
  • Latency — end-to-end time from prompt to completed response.
  • Score — the LLM-judge score, colored green when the case passed and red when it failed.
Click a row to expand its drawer. The drawer shows the full evidence for the case:
  • Expected response topic — what the golden set says a correct answer covers. Compare this against the actual response before you trust the judge’s score either way.
  • Actual response — the agent’s reply, verbatim.
  • Transcript — the caller prompt as the STT leg heard it. An empty or garbled transcript means the problem started upstream of the LLM.
  • Judge reasoning — the LLM judge’s justification for its score. Read this before promoting a candidate: a passing score with thin reasoning is not a rollout signal.
  • Recording — the case’s audio, playable inline.
  • Failure reason — when a case failed to execute (not merely failed its score), the reason appears here. see section 6 for how failure text is presented safely.
While the run is in Pending or Running, the table shows No results yet — run is still processing; it fills in case-by-case as the runner finishes each one. A completed run with an empty cases table means the golden set was empty or every case was skipped — check the golden set before rerunning.

​
4. Per-leg latency tiles

Each expanded case splits its end-to-end latency into four tiles: Distinguishing the legs matters: an STT failure shows a missing or garbled transcript with a low score, but healthy LLM latency — routing blame to transcription quality. An LLM slowdown shows a clean transcript and a correct-but-slow response, with the LLM tile carrying the bulk of the end-to-end number. Tuning them is different work.

​
5. The Simulated badge and the stub warning

In the current rollout, the STT and TTS latency legs are stubbed — the API flags every run with is_simulated: true and the dashboard attaches a Simulated pill to the header plus an explanatory notice below it. The rule: never treat simulated-run STT/TTS latency numbers as procurement-grade. The LLM leg and the end-to-end figure contain real signal; the STT/TTS tiles are placeholder values until those legs are executed for real. Quotes, SLAs, or provider comparisons must come from a run against live providers, not from a simulated run. The API response carries the same flag (is_simulated, simulated_legs, simulation_notice) so automation can enforce the same rule — filter simulated runs out of any numbers shipped to a vendor or regulator.

​
6. The run-level failure banner

When an entire run faults — a case-row write error, an unexpected runner fault, a crash mid-run — the runner persists a synthetic run-level failure case whose golden case id is __run__. The dashboard surfaces that diagnostic as a red This run failed banner on the header card and keeps the __run__ case out of the per-case results table (it is not a golden case; it carries no transcript or score). Reading the banner:
  • Run failed, no case detail — the diagnostic shows the run-level cause, for example a runner fault or an infrastructure error.
  • Run failed mid-cases — the banner falls back to the first per-case failure reason, so a failed run always names a cause rather than showing a bare red badge.
  • Failure text is redacted — provider and SDK error messages can carry stack traces, internal hostnames, IP addresses, file paths, and long credential-shaped tokens. Before rendering, the page strips those out (paths, IPs, cluster hostnames, hex blobs, and UUIDs are replaced with placeholders, and the text is truncated to a safe length). When you are diagnosing a failure through the API instead, apply the same treatment before pasting error text into tickets or chat.

​
7. API parity

Everything on the page comes from one endpoint, so any view on the dashboard can be reproduced — or exported — over the API:
The response returns the run object (status, cases_total, cases_passed, p95_latency_ms, cost_usd, stt_provider, tts_provider, llm_model, is_simulated, simulated_legs, simulation_notice) plus the full case array with transcripts, judge scores, and per-leg latencies. The run-level failure diagnostic is the case whose golden_case_id equals __run__. Useful companions:
  • GET /api/v1/agents/voice-eval/runs/{id}/regression-delta — compares a run against its prior baseline run for the same agent + golden set and reports whether P95 latency or judge score regressed.
  • GET /api/v1/agents/voice-eval/runs?status=failed — monitor for failed runs across all agents.

​
See also