​

Voice Agent Quality Index (VAQI)

​

What VAQI measures

​

Latency percentiles

• TTFB (time to first byte) — per session, the average time from the end of a caller’s turn to the first audio byte the agent speaks. The card shows the mean across sessions and the p95 across per-session averages, so one bad session cannot hide inside the mean. Callers read a pause over roughly a second as a dead line; a rising p95 is the earliest sign of an LLM or STT slowdown.
• Turn gap — the average silence between turns within a session (the per-turn counterpart of TTFB). Again mean plus p95. The dashboard highlights the card when p95 exceeds 1500 ms (TTFB warn threshold is 1200 ms).

​

Turn-taking and barge-in events

• Turns per session — the average number of caller↔agent exchanges. Longer sessions with healthy turn counts mean the agent is actually conversing; a drop can mean the agent hangs up early or callers abandon.
• Barge-in rate — the share of turns where the caller interrupted the agent’s speech. Some barge-in is healthy — callers who cannot interrupt experience the agent as a talking wall. A rising rate after a prompt change usually means the agent is too verbose or answers without pausing.
• Barge-in success rate (clean yield) — of all interruptions, the share where the agent stopped cleanly and let the caller take the turn. A low success rate means the agent talks over interruptions and produces the double-speak overlap that callers describe as “the bot wouldn’t let me speak.”

​

Where to find it and how to pick the time window

​

Reading the breakdown and setting thresholds

• Sample size first. A day with 3 sessions averaging 900 ms is noise; a day with 400 sessions averaging 900 ms is a regression. The table gives you the per-day denominator.
• Compare like-for-like weekdays. A Monday that lands above the previous Monday’s turn gap is a signal; a Monday above a Saturday is traffic-mix, not a problem.

​

Wire a VAQI regression into KPI alerts

1. Open Insights → Alerts (/insights/alerts) and click Create rule.
2. Pick the metric (containment for mechanical regressions, CSAT for caller-felt quality).
3. For a known number, use a threshold rule — e.g. containment below 70% over a 7d window. If you do not know the right number yet, use anomaly mode, which flags a drop against the metric’s own learned baseline.
4. The fired notification deep-links into the metric page; from there, VAQI tells you whether the root cause is mechanical (latency, turn-taking) or content (prompt, routing).

​

Worked example: a latency regression after a model swap

1. Sessions card shows ~1,900 sessions in the window — the aggregates are trustworthy.
2. TTFB card reads 890 ms avg / 1.9 s p95, highlighted above the 1200 ms warn threshold. Previously it sat at 620 ms / 1.1 s.
3. The component cards isolate it: avg LLM latency moved from 410 ms to 780 ms, while STT and TTS stayed flat. The cheaper model is slower to first token.
4. The daily breakdown confirms the anchor: turn gap climbs starting Monday, matching the deployment — a step change, not gradual drift.
5. Barge-in rate ticked up from 12% to 19% — callers started interrupting the slower-speaking agent — while barge-in success held at 91%, so at least interruptions yield cleanly.
6. The fix decision is now a trade-off with numbers on both sides: keep the cheaper model and accept ~1.9 s p95 (below the 2 s threshold the team set for this use case), or revert to the faster model and treat the delta as the price of the upgrade path. A threshold alert on containment catches the caller-visible effect if the team keeps the swap.

​

Analytics API and permissions

curl -H "x-api-key: $ORBIT_API_KEY" \
  "https://orbit.devotel.io/api/v1/voice/vaqi?days=7"

{
  "data": {
    "days": 7,
    "sessions": 1923,
    "avg_session_duration_ms": 132000,
    "avg_turns_per_session": 8.4,
    "avg_turn_gap_ms": 640,
    "p95_turn_gap_ms": 1180,
    "avg_ttfb_ms": 620,
    "p95_ttfb_ms": 1090,
    "avg_stt_ms": 180,
    "avg_llm_ms": 410,
    "avg_tts_ms": 220,
    "barge_in_rate": 12.4,
    "barge_in_success_rate": 90.8,
    "daily": [
      { "date": "2026-08-27", "sessions": 288, "avg_turn_gap_ms": 610, "avg_ttfb_ms": 590 }
    ]
  }
}

• Dashboard route — restricted to owner, admin, and developer roles, the same gate as the rest of voice settings and quality signals. Other roles see no page.
• API — a valid API key or JWT on the voice reads surface. Rate-limited like the other voice read endpoints.
• Alert rules — reads need the analytics read scope; creating, updating, or deleting a rule additionally requires owner, admin, or developer, and each change is recorded in the audit log.

​

Troubleshooting

• “No voice AI sessions yet for this period.” No completed voice-agent sessions fall in the selected window. Widen to 30d; metrics appear as soon as the voice gateway completes a session.
• Page fails to load. The page surfaces the error with a retry action rather than an infinite spinner, and sessions keep recording — it is a dashboard-only outage. If it persists, check status.orbit.devotel.io.
• Averages look fine but callers complain. Read p95, not the mean; the warn highlight fires on p95 exactly because averages hide the tail.

​

See also

• Read the Voice → Intelligence page — the content counterpart: what callers said, not how fast the line answered
• KPI alerts on business metrics — threshold and anomaly rules for containment, CSAT, and LLM spend