Skip to main content

​
The call-quality evaluation lifecycle

Quality management has two halves. The first is the scorecard pipeline — evaluation forms, reviewer scorecards, acknowledge/appeal, AI auto-scoring, and the leaderboard — covered in QA evaluations and the performance leaderboard. The second half, covered here, is the operations layer around those scorecards: routing review work to humans, keeping reviewers consistent with each other, turning QA output into coaching, verifying that auto-scoring is actually covering your traffic, and rolling per-call scores up into org-wide summary, trends, drivers, and custom supervisor composites. A call finishes, a recording and transcript land in the library, and then one of two things happens: the AI auto-scores it, or a supervisor assigns it to a human evaluator as a scoring task. Everything below is what happens to that scoring task — from assignment through calibration to the rollups supervisors watch.

​
The cast of entities

  • Evaluation assignment — a routing envelope for a scoring task, not a score. Assigning picks a call and names a reviewer; the row later carries the reviewer’s scorecard. An assignment stays open until the assigned reviewer scores the call, so open-assignment count is the workload measure. Supervisors rebalance open work between reviewers, every reviewer holds a quota of open assignments (the API rejects over-quota assigns with 409 QUOTA_EXCEEDED), and a call that already has an open assignment cannot be double-assigned (409 ALREADY_ASSIGNED). Due dates are derived from your tenant’s assignment window (created_at + due_days), not stamped on the row — change the window in settings and the workload view recomputes.
  • Calibration session — a scheduled inter-rater agreement pass that pins one call and one form and invites a roster of reviewers. Every invited reviewer scores the call blind: they cannot read another reviewer’s score — or the AI auto-score — until they submit their own, and that rule is enforced server-side, not just hidden in the UI. Once at least two reviewers submit, the session report shows per-criterion variance, overall agreement, and each reviewer’s delta from group consensus — including the AI’s drift from that consensus. Calibration scores are practice rows: they never enter official averages, the leaderboard, or the agent’s acknowledge/appeal flow.
  • Coaching cards — per-agent readouts folded from official reviewer-authored evaluations: average QA score, flagged share, weakest criteria, plus short badge chips. Supervisors read them as a “who needs drilling” view; agents read their own card scoped to themselves.
  • Autoscore coverage — the one-glance answer to “how much of my traffic is the automated AI scorecard actually scoring?” It counts completed calls in the trailing window against calls carrying an AI-scored evaluation, across both the voice scorecard lane and the conversation-judge lane, and averages the automated scores.
  • Auto-QA readiness — the operational health snapshot for the automation: is the auto-scoring toggle on, does an active evaluation form exist, are eligible calls landing in the window, and what share of those candidates actually came back with an AI scorecard. Coverage tells you how much; readiness tells you why not when the number is wrong — a disabled toggle, no active form, or calls without transcripts.

​
Scorecard vocabulary

Three views read the same official-score contract — graded, reviewer-authored evaluations only (an agent scoring their own call, an un-scored queued placeholder, and rows under appeal never enter an official average):
  • The individual call scorecard — one weighted evaluation of one call against one form. Authored by a human reviewer or by the AI auto-scorer, correctable by override, and acknowledged or appealed by the agent. This is the unit everything else aggregates.
  • The supervisor scorecard — a per-agent 0–100 composite you define on the fly: custom KPIs (QA average, average handle time, calls handled, CSAT, queue SLA, first-contact resolution) with your own weights and targets, weighted across voice and QA into one number. The definition comes in per request — nothing is stored; the composite is a read-time rollup over the same tables.
  • QM drivers — the per-criterion rollup: every scorecard line (“Greeting”, “Policy disclosure”, “Resolution”) normalised to a 0–100 average and sorted worst-first, so “which lines move the score” is a read, not a spreadsheet.

​
The lifecycle

  1. Open. A supervisor opens an assignment — hands a specific call to a specific reviewer — or samples work automatically: a form with a sample_rate_pct above zero queues a share of each agent’s calls for human review, and with auto-scoring enabled every completed call is AI-scored in the background. Assignments respect per-reviewer quotas, and an open assignment can be reassigned when workloads shift.
  2. Calibrate. Calibration sessions sit alongside production scoring as a consistency pass — reviewers score the same call blind, and the session report exposes drift per reviewer, per criterion, and for the AI against group consensus. A team whose reviewers disagree on the calibration call will disagree on live ones; fix the rubric or the coaching, then re-run.
  3. Score and react. The assigned reviewer scores the call; the agent acknowledges or appeals; a supervisor resolves. Low scores auto-generate coaching assignments, and coaching cards accumulate the per-agent picture.
  4. Roll up. The supervisor-level summary folds both scoring pipelines — reviewer-authored call scorecards and the AI-judged conversation scores on digital channels — into one org-wide view: pass rate, average confidence, breakdowns by channel and rubric, plus a recent-failures triage list. Trends give the per-day weighted-score average per agent and per form. Drivers rank the criteria. The gamification leaderboard ranks agents on the tuned points mix. Supervisors who want one number build a composite scorecard over the same facts.
  5. Verify coverage. Autoscore coverage and auto-QA readiness close the loop on automation itself: what share of traffic the AI scored, what it averaged, and — when coverage sags — which generation gate is off.

​
Where reads sit, where writes are gated

Every surface above is mounted under /api/v1/quality. The split:
  • Writes (reviewer scope — owner, admin, or supervisor): assignment and reassignment, calibration sessions and blind scores, composite-scorecard definitions, leaderboard computation, and the scorecard lifecycle itself (author, override, resolve).
  • Reads: workload, session reports, coaching cards, compliance flags, coverage, readiness, summary, trends, and drivers are all GETs any reviewer can run; agents read only their own card and flags through self-scoped guards.
  • Blind-gated reads: a calibration report returns 403 to an invited reviewer who has not submitted their own score yet — the blind rule applies to reading agreement, too.
Key routes: The recording library is the worklist all of this draws from: every recording in the tenant, full-text-searchable over finalised diarised transcripts, joined to the linked QA evaluation score and the recording QC verdict — so “recordings where the customer said X, scored below Y, QC-failed” is one query, reviewer-scoped because it surfaces transcript content across all agents. The full request/response shapes are in the Quality API reference; the per-call scorecard pipeline itself is documented in QA evaluations and the performance leaderboard.