Skip to main content

​
QA Evaluations: scorecards and the appeal lifecycle

The Quality → Evaluations page is the manual, human-graded QA pipeline: a supervisor authors a weighted scorecard form, reviewers grade conversations against it, the evaluated agent acknowledges or appeals the score, and a supervisor resolves the appeal. This page walks through building a form, grading a conversation, and what each evaluation status moves through — including how AI auto-scoring and agent self-evaluations feed the same pipeline. What you see depends on your role. Reviewers (owner, admin, supervisor) get five tabs — Evaluations, Auto-scored, Workload, Calibration, and Scorecards. Agents get a single My evaluations view — the list is scoped to their own scores on the server, not just hidden in the UI — with acknowledge and appeal actions and an Evaluate my call self-assessment shortcut.

​
Create a scorecard form

The Scorecards tab lists every form with its section count, criterion count, and auto-sample rate. If you already have forms, a search box filters the list by name.
  1. Open Quality → Evaluations → Scorecards and choose New scorecard.
  2. Name the form and add an optional description — for example, “Inbound support QA”.
  3. Build the form as sections containing criteria. A section carries a label and a numeric weight; each criterion carries a label, a numeric weight, and a max score (1–100, defaulting to 100).
  4. Toggle Auto-fail on any section where a zero must fail the whole interaction — the classic compliance pattern, like “identity verification” or “PII handling”. If any criterion in an auto-fail section scores zero, the total is forced to zero.
  5. Optionally set an auto-sample rate (0–100%). A rate above zero auto-queues that share of each agent’s conversations as pending rows for reviewers to claim and score. Set it to 0 for manual-only sampling, and manage the rate afterward from the Workload tab.
  6. Create scorecard. Activate or deactivate it from the list — an inactive form is no longer offered in the grading dropdown.
Edit an existing form to retune weights or add criteria; the weighted total is calculated by the server on every save, so browser-side arithmetic can never rewrite a score.

​
Score a conversation

Open the Evaluations tab and choose Score a conversation. If no active form exists yet, the button points you back to the Scorecards tab.
  1. Pick the scorecard, then enter the agent id and call id you’re grading.
  2. Rate each criterion on its 0–max scale. A live weighted-total preview shows the score so far; the row turns red while it’s zero.
  3. Save evaluation. The weighted total is re-derived on the server and written along with the raw scores.
Every row links out to the underlying call recording and transcript — open it from the phone icon on the row and verify a criterion score against what was actually said, rather than taking the number on faith. Rows paginate 25 at a time behind Load more, so a month of heavy scoring doesn’t drag the page down. Filter the list by status or agent (typing pauses before it refetches), or flip on CSAT-triggered only to isolate rows the low-CSAT trigger queued. Reviewers also see a Variance action on each reviewed row — it compares the agent’s self-evaluation against the reviewer score for the same call, per criterion.

​
Statuses: pending → acknowledged → appealed → resolved

Every evaluation moves through a four-status lifecycle, and the actions available change at each step:
  • Pending — a fresh score the agent hasn’t reacted to. The agent can Acknowledge it or Appeal it with a written note. Pending rows that came from the auto-sampler or the low-CSAT trigger instead show a Claim & score action for reviewers, and an open, still-untouched human assignment shows Reassign to move it to a different evaluator.
  • Acknowledged — the agent has accepted the score. An appeal is still possible while the status is pending or acknowledged.
  • Appealed — the agent sent a written appeal note; reviewers see a Resolve action on the row.
  • Resolved — the supervisor reviewed the appeal and closed it with an optional resolution note. Resolving does not regrade the score — it records that the appeal was reviewed.
Self-evaluations (the agent’s Evaluate my call) land pre-acknowledged with a Self badge, are excluded from the official QA average, and skip the appeal loop entirely — there is no reviewer to appeal to on a self-score.

​
AI auto-scoring with human override

The Auto-scored tab lists evaluations the AI auto-score pass wrote against your active form. A background sweep grades every completed conversation that has a transcript; rows scoring under your flag threshold (70 out of 100 by default) get a flagged badge so the supervisor’s review queue carries only the borderline conversations. A Flagged for review only switch narrows the list to those rows. If the tab shows “AI Auto-QA is off”, enable it under Settings → AI Auto-QA — the sweep skips tenants that haven’t opted in, so nothing is scored until you turn it on. Each pending AI-scored row offers Override: a reviewer grades the same criteria and saves, and the server re-derives the weighted total from the form’s weights — keeping an AI score correctable in place instead of creating a shadow copy.

​
Self-evaluations and the variance view

An agent scores their own conversation with Evaluate my call: pick the scorecard and call id, grade yourself, and save. Self-evaluations are excluded from the official QA average by default — an agent can’t inflate their own ranking by scoring themselves — they’re for reflection, and they feed calibration. Both reviewers and agents can open the Variance dialog on a row. It pairs the agent’s self-evaluation with the reviewer’s score for the same call and shows the total plus a per-criterion delta, so a coaching conversation starts from where the two readings diverge. The Workload tab shows every eligible evaluator with their open-assignment count against the tenant’s quota, remaining capacity, and how many of their open assignments are overdue — both values are always server-derived. From here you assign a specific conversation to a specific evaluator and set per-agent quotas and due dates. Reassigning an open assignment from the Evaluations tab rebalances without re-scoring. For the org-wide trend view, the parent Quality dashboard charts per-day average scores across the official evaluations (reviewer-authored plus AI auto-scored), with breakdowns by agent and by scorecard form — the “is quality moving” read. Calibration on this page runs blind inter-rater sessions: a supervisor opens a session on one conversation, each reviewer scores without seeing the others’ results, and the report compares each reviewer’s delta from consensus — including how far the AI auto-score drifts from the human consensus. Keep the Appealed status filter in view during the week: it isolates the queue of written agent appeals waiting on a resolution.

​
How this differs from the LLM auto-score

Both pipelines write into the same evaluations list, and an agent can acknowledge or appeal either one. The difference is provenance: a manual evaluation carries the reviewer’s identity, while an AI evaluation is flagged as auto-scored so downstream readers and your own percent-auto-scored reports can tell the two apart. Auto-scored conversations count toward official averages the same way human ones do, and the reviewer-override path corrects them in place. If you’re deciding between them: leave auto-scoring on for 100% coverage, and spend reviewer time spot-auditing the flagged rows plus a claimed share of sampled conversations rather than grading every conversation from scratch.

​
API surface

Everything the page does is backed by the Quality API, so the same lifecycle runs from scripts and QA tooling. Reviewer-scope routes (owner / admin / supervisor) handle forms, scoring, overrides, and resolutions; agent-scope routes handle acknowledge, appeal, and self-evaluation.

​
Roles and eligibility

Reviewer-scope routes (forms, scoring, resolution, claim-score, override) accept the owner, admin, or supervisor role with the inbox:read / inbox:write scope. Agent-scope routes (acknowledge, appeal, self-evaluation, list, variance) accept any authenticated caller with the inbox:read / inbox:write scope, and every agent route enforces ownership server-side: the acknowledge and appeal handlers reject any caller that isn’t the evaluated agent (403), the self-evaluation route stamps the caller as both reviewer and agent, and a non-reviewer listing or variance request is forced to the caller’s own rows. The samples below authenticate with X-API-Key like every other Orbit API call; the base URL is https://api.orbit.devotel.io/api/v1.

​
Create an evaluation (reviewer scoring)

Post the agent id, the call id, and per-criterion raw scores keyed by criterion id, each on a 0–100 scale. The server re-derives the weighted total_score from the form’s weights, so the rollup the POST returns is never client-supplied. The row lands pending for the agent to acknowledge or appeal.
cURL
The SDK escape hatch (orbit.request, covered in API recipes) reaches any REST path a typed method does not wrap:
Node SDK
The 201 response echoes the stored row, with total_score and the timestamps server-derived:
Two failures to expect before the row writes: a 404 NOT_FOUND when the form_id doesn’t exist, and a 409 FORM_INACTIVE when the form has been deactivated — reactivate it from the Scorecards tab, then retry.

​
Create a self-evaluation (agent scoring their own call)

The self-evaluation body intentionally has no agent_id: the handler stamps the caller as both reviewer and agent, so an agent can only ever score their own work. The row lands acknowledged — there is no reviewer to appeal to on a self-score — and carries evaluation_type: "self" so it stays out of the official QA average.
cURL
Node SDK
The 201 response lands pre-acknowledged with the derived type on the row:

​
Acknowledge, appeal, resolve, claim-score, and override

All five actions share the same id route — substitute the action name for {action}:
cURL
The actions differ only in who may call them and what the body carries: An appeal is one POST with the written note:
Node SDK
The response is the moved row’s new status plus the stamped timestamps:
Calling an action in the wrong state — appealing an appealed row, resolving a pending one, overriding a human-authored score — returns a 409 INVALID_STATE; a caller who isn’t the evaluated agent gets a 403 FORBIDDEN.

​
List evaluations (filters and keyset pagination)

Reviewer scope filters by agent_id, call_id, reviewer_id, status (pending, acknowledged, appealed, resolved), csat_triggered=true (rows queued by a detractor CSAT), auto_scored=true (AI-authored rows), flagged_for_review=true (below-threshold auto-scores waiting on a human), and evaluation_type=self|reviewer. A non-reviewer caller only ever sees their own evaluations — the agent_id filter is forced to the caller on the server. Pages are keyset-paginated: read next_cursor out of the response, then pass it back as cursor. A null next_cursor ends the walk, and paging is O(limit) at any depth — no offset drift.
cURL
Node SDK
Each item carries the derived evaluation_type and any linked CSAT score for the same call; the page ends with the opaque keyset cursor to pass back:
A malformed cursor returns a 400 INVALID_CURSOR — refetch from the beginning rather than guessing at the cursor’s format. The csat_score fields are null on evaluations with no linked survey response, which is most reviewer-authored rows.

​
Compare self vs reviewer scores (variance)

Pass the call_id (required). Reviewer scope also passes agent_id to inspect any agent; a non-reviewer caller is forced to their own variance and gets a 403 for another agent’s id. When both sides exist, the variance block carries the total delta plus one entry per criterion — the starting point for a coaching conversation.
cURL
Node SDK
self or reviewer is null — and so is variance — until both a self-evaluation and a reviewer score exist for the call. Check for null before charting the delta.

​
See also