Skip to main content

​
QA Evaluations: scorecards and the appeal lifecycle

The Quality → Evaluations page is the manual, human-graded QA pipeline: a supervisor authors a weighted scorecard form, reviewers grade conversations against it, the evaluated agent acknowledges or appeals the score, and a supervisor resolves the appeal. This page walks through building a form, grading a conversation, and what each evaluation status moves through — including how AI auto-scoring and agent self-evaluations feed the same pipeline. What you see depends on your role. Reviewers (owner, admin, supervisor) get five tabs — Evaluations, Auto-scored, Workload, Calibration, and Scorecards. Agents get a single My evaluations view — the list is scoped to their own scores on the server, not just hidden in the UI — with acknowledge and appeal actions and an Evaluate my call self-assessment shortcut.

​
Create a scorecard form

The Scorecards tab lists every form with its section count, criterion count, and auto-sample rate. If you already have forms, a search box filters the list by name.
  1. Open Quality → Evaluations → Scorecards and choose New scorecard.
  2. Name the form and add an optional description — for example, “Inbound support QA”.
  3. Build the form as sections containing criteria. A section carries a label and a numeric weight; each criterion carries a label, a numeric weight, and a max score (1–100, defaulting to 100).
  4. Toggle Auto-fail on any section where a zero must fail the whole interaction — the classic compliance pattern, like “identity verification” or “PII handling”. If any criterion in an auto-fail section scores zero, the total is forced to zero.
  5. Optionally set an auto-sample rate (0–100%). A rate above zero auto-queues that share of each agent’s conversations as pending rows for reviewers to claim and score. Set it to 0 for manual-only sampling, and manage the rate afterward from the Workload tab.
  6. Create scorecard. Activate or deactivate it from the list — an inactive form is no longer offered in the grading dropdown.
Edit an existing form to retune weights or add criteria; the weighted total is calculated by the server on every save, so browser-side arithmetic can never rewrite a score.

​
Score a conversation

Open the Evaluations tab and choose Score a conversation. If no active form exists yet, the button points you back to the Scorecards tab.
  1. Pick the scorecard, then enter the agent id and call id you’re grading.
  2. Rate each criterion on its 0–max scale. A live weighted-total preview shows the score so far; the row turns red while it’s zero.
  3. Save evaluation. The weighted total is re-derived on the server and written along with the raw scores.
Every row links out to the underlying call recording and transcript — open it from the phone icon on the row and verify a criterion score against what was actually said, rather than taking the number on faith. Rows paginate 25 at a time behind Load more, so a month of heavy scoring doesn’t drag the page down. Filter the list by status or agent (typing pauses before it refetches), or flip on CSAT-triggered only to isolate rows the low-CSAT trigger queued. Reviewers also see a Variance action on each reviewed row — it compares the agent’s self-evaluation against the reviewer score for the same call, per criterion.

​
Statuses: pending → acknowledged → appealed → resolved

Every evaluation moves through a four-status lifecycle, and the actions available change at each step:
  • Pending — a fresh score the agent hasn’t reacted to. The agent can Acknowledge it or Appeal it with a written note. Pending rows that came from the auto-sampler or the low-CSAT trigger instead show a Claim & score action for reviewers, and an open, still-untouched human assignment shows Reassign to move it to a different evaluator.
  • Acknowledged — the agent has accepted the score. An appeal is still possible while the status is pending or acknowledged.
  • Appealed — the agent sent a written appeal note; reviewers see a Resolve action on the row.
  • Resolved — the supervisor reviewed the appeal and closed it with an optional resolution note. Resolving does not regrade the score — it records that the appeal was reviewed.
Self-evaluations (the agent’s Evaluate my call) land pre-acknowledged with a Self badge, are excluded from the official QA average, and skip the appeal loop entirely — there is no reviewer to appeal to on a self-score.

​
AI auto-scoring with human override

The Auto-scored tab lists evaluations the AI auto-score pass wrote against your active form. A background sweep grades every completed conversation that has a transcript; rows scoring under your flag threshold (70 out of 100 by default) get a flagged badge so the supervisor’s review queue carries only the borderline conversations. A Flagged for review only switch narrows the list to those rows. If the tab shows “AI Auto-QA is off”, enable it under Settings → AI Auto-QA — the sweep skips tenants that haven’t opted in, so nothing is scored until you turn it on. Each pending AI-scored row offers Override: a reviewer grades the same criteria and saves, and the server re-derives the weighted total from the form’s weights — keeping an AI score correctable in place instead of creating a shadow copy.

​
Self-evaluations and the variance view

An agent scores their own conversation with Evaluate my call: pick the scorecard and call id, grade yourself, and save. Self-evaluations are excluded from the official QA average by default — an agent can’t inflate their own ranking by scoring themselves — they’re for reflection, and they feed calibration. Both reviewers and agents can open the Variance dialog on a row. It pairs the agent’s self-evaluation with the reviewer’s score for the same call and shows the total plus a per-criterion delta, so a coaching conversation starts from where the two readings diverge. The Workload tab shows every eligible evaluator with their open-assignment count against the tenant’s quota, remaining capacity, and how many of their open assignments are overdue — both values are always server-derived. From here you assign a specific conversation to a specific evaluator and set per-agent quotas and due dates. Reassigning an open assignment from the Evaluations tab rebalances without re-scoring. For the org-wide trend view, the parent Quality dashboard charts per-day average scores across the official evaluations (reviewer-authored plus AI auto-scored), with breakdowns by agent and by scorecard form — the “is quality moving” read. Calibration on this page runs blind inter-rater sessions: a supervisor opens a session on one conversation, each reviewer scores without seeing the others’ results, and the report compares each reviewer’s delta from consensus — including how far the AI auto-score drifts from the human consensus. Keep the Appealed status filter in view during the week: it isolates the queue of written agent appeals waiting on a resolution.

​
How this differs from the LLM auto-score

Both pipelines write into the same evaluations list, and an agent can acknowledge or appeal either one. The difference is provenance: a manual evaluation carries the reviewer’s identity, while an AI evaluation is flagged as auto-scored so downstream readers and your own percent-auto-scored reports can tell the two apart. Auto-scored conversations count toward official averages the same way human ones do, and the reviewer-override path corrects them in place. If you’re deciding between them: leave auto-scoring on for 100% coverage, and spend reviewer time spot-auditing the flagged rows plus a claimed share of sampled conversations rather than grading every conversation from scratch.

​
API surface

Everything the page does is backed by the Quality API, so the same lifecycle runs from scripts and QA tooling. Reviewer-scope routes (owner / admin / supervisor) handle forms, scoring, overrides, and resolutions; agent-scope routes handle acknowledge, appeal, and self-evaluation.

​
See also