Skip to main content

​
Voice Agent Quality Index (VAQI)

An AI voice agent can pass every offline eval and still feel slow to live callers. VAQI is the live-session complement to evals: it aggregates per-session quality records your voice agents emit on every call and rolls them up into latency, turn-taking, and barge-in metrics over a time window you pick — 24 hours, 7 days, or 30 days. Use it to answer “how is the fleet doing right now,” not “does this agent build pass its tests.” Base path: /api/v1/voice Authentication: Clerk session (Authorization: Bearer <token>) or API key (X-API-Key). Scope: voice read. Dashboard page: Voice → Voice Agent Quality (available to owners, admins, and developers).

​
1. What VAQI measures

Every completed AI voice session produces one record: duration, turn count, the time the agent took to start responding, the silence between caller and agent turns, speech-to-text / LLM / text-to-speech processing time, and how interruption (barge-in) behaved. VAQI returns:
  • sessions — sessions in the window
  • avg_session_duration_ms / avg_turns_per_session
  • avg_ttfb_ms, p95_ttfb_ms — time-to-first-byte: how fast the agent starts speaking after the caller’s turn ends
  • avg_turn_gap_ms, p95_turn_gap_ms — the silent gap between turns across the session
  • avg_stt_ms, avg_llm_ms, avg_tts_ms — pipeline stage averages
  • barge_in_rate — % of turns where the caller interrupted the agent
  • barge_in_success_rate — % of those interruptions where the agent yielded cleanly
  • daily[] — per-day breakdown of sessions, average turn gap, and average TTFB
An empty window returns zeros (not an error), and the fall-back when the rollup store is unavailable is the same zero shape — so a healthy-but-quiet tenant and a storage hiccup both read as “no data,” never as broken metrics.

​
2. Reading the three KPI families

Latency (ms) — TTFB is the number callers feel first. The dashboard flags a warning when the p95 exceeds 1,200 ms. Splitting the pipeline (STT vs LLM vs TTS) tells you which stage drifted: a TTFB spike with a flat STT and TTS but an elevated LLM average points at the model, not the speech pipeline. Turn-taking (turns + gaps) — avg_turns_per_session shows whether conversations are real exchanges, and the turn gap measures the dead air between them. The dashboard warns when the p95 turn gap exceeds 1,500 ms. A rising gap with a stable TTFB usually means longer generations later in the session, so check your prompt length and tool call sizes. Barge-in (interruptions) — barge_in_rate is the share of turns the caller talks over the agent; barge_in_success_rate is the share of those where the agent stopped cleanly. Some interruption is healthy — callers probe and correct. A high rate with a low success rate means callers are fighting a talking-over agent. Example thresholds that match the dashboard warning markers:

​
3. Picking a window and comparing agents

Choose the narrowest window with enough sessions to make the averages meaningful:
  • 24h while tuning a prompt or model change.
  • 7d for the operating baseline (the default).
  • 30d for trend review.
The rollup covers all voice agents in the tenant. To compare two agents, run each in its own window and read the daily[] rows across the days each was live, or route each agent through a comparison window of its own for a clean read. The dashboard page refreshes every 30 seconds, so a shared screen on the 24h window works as a live wallboard.

​
4. Alert thresholds — and how VAQI differs from evals

Wire alerts on the two p95 warnings above; filter small-sample noise by only alerting when the window has enough sessions. Evals and VAQI answer different questions, and you need both:
  • Evals run offline against recorded or scripted conversations — they answer “did the agent say the right thing.”
  • VAQI is computed from live sessions — it answers “does the agent feel fast and polite to real callers right now.”
Treat a failing eval as a correctness gate and a VAQI p95 breach as an operations alert.

​
5. Who reads it

  • Operations watches p95 TTFB and turn gap and owns the alert thresholds.
  • Agent builders own the pipeline split (STT/LLM/TTS) and barge-in behavior — a regression maps back to a prompt, model, or voice configuration change.

​
6. Troubleshooting

  • Latency spikes after a deploy: compare the stage averages against your baseline. If the LLM average moved, the prompt or the model changed; if STT/TTS moved, the speech configuration changed.
  • Cold models: the first sessions of the day can carry a higher TTFB while the speech and model pipeline warms; widen to the 7d window before you treat it as a regression, and read daily[] to see whether today’s row is the only offender.
  • High barge-in with low success: the agent is not yielding — shorten end-of-turn sensitivity or the first response length.
  • All zeros: either no AI voice sessions completed in the window, or the rollup cache was briefly unavailable. If call volume is running but VAQI still returns zeros, retry — the fall-back is deliberately zero-shaped so the page renders.

​
Worked example

GET /api/v1/voice/vaqi?days=1
Reading it: LLM time dominates TTFB (388 of 742 ms), p95 TTFB is under the 1,200 ms warning marker, and a 6.8% interruption rate with a 91% clean-yield rate is healthy.

​
What the dashboard panel shows

Voice → Voice Agent Quality renders the same rollup: a sessions, duration, TTFB (avg/p95), and turn-gap (avg/p95) KPI row that highlights in a warning colour past the thresholds; a pipeline split (STT / LLM / TTS); barge-in rate and barge-in success cards; and the per-day breakdown table. A window switcher (24h / 7d / 30d) is part of the URL, so you can deep-link a specific window.
Related: AI agent evals · Call QA scorecards · AI agent outbound calls