> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# QA Evaluations: weighted scorecards, appeals, and the grading workflow

> Build weighted evaluation forms, score conversations, and run the acknowledge/appeal/resolve lifecycle — the manual, human-graded QA pipeline under Quality → Evaluations, plus where AI auto-scoring and agent self-evaluations fit.

# QA Evaluations: scorecards and the appeal lifecycle

The **Quality → Evaluations** page is the manual, human-graded QA pipeline: a supervisor authors a weighted scorecard form, reviewers grade conversations against it, the evaluated agent acknowledges or appeals the score, and a supervisor resolves the appeal. This page walks through building a form, grading a conversation, and what each evaluation status moves through — including how AI auto-scoring and agent self-evaluations feed the same pipeline.

What you see depends on your role. Reviewers (owner, admin, supervisor) get five tabs — **Evaluations**, **Auto-scored**, **Workload**, **Calibration**, and **Scorecards**. Agents get a single **My evaluations** view — the list is scoped to their own scores on the server, not just hidden in the UI — with acknowledge and appeal actions and an **Evaluate my call** self-assessment shortcut.

## Create a scorecard form

The **Scorecards** tab lists every form with its section count, criterion count, and auto-sample rate. If you already have forms, a search box filters the list by name.

1. Open **Quality → Evaluations → Scorecards** and choose **New scorecard**.
2. Name the form and add an optional description — for example, "Inbound support QA".
3. Build the form as **sections** containing **criteria**. A section carries a label and a numeric **weight**; each criterion carries a label, a numeric **weight**, and a **max score** (1–100, defaulting to 100).
4. Toggle **Auto-fail** on any section where a zero must fail the whole interaction — the classic compliance pattern, like "identity verification" or "PII handling". If any criterion in an auto-fail section scores zero, the total is forced to zero.
5. Optionally set an **auto-sample rate** (0–100%). A rate above zero auto-queues that share of each agent's conversations as pending rows for reviewers to claim and score. Set it to 0 for manual-only sampling, and manage the rate afterward from the **Workload** tab.
6. **Create scorecard.** Activate or deactivate it from the list — an inactive form is no longer offered in the grading dropdown.

Edit an existing form to retune weights or add criteria; the weighted total is calculated by the server on every save, so browser-side arithmetic can never rewrite a score.

## Score a conversation

Open the **Evaluations** tab and choose **Score a conversation**. If no active form exists yet, the button points you back to the Scorecards tab.

1. Pick the **scorecard**, then enter the **agent id** and **call id** you're grading.
2. Rate each criterion on its **0–max** scale. A live weighted-total preview shows the score so far; the row turns red while it's zero.
3. **Save evaluation.** The weighted total is re-derived on the server and written along with the raw scores.

Every row links out to the underlying call recording and transcript — open it from the phone icon on the row and verify a criterion score against what was actually said, rather than taking the number on faith. Rows paginate 25 at a time behind **Load more**, so a month of heavy scoring doesn't drag the page down.

Filter the list by **status** or **agent** (typing pauses before it refetches), or flip on **CSAT-triggered only** to isolate rows the low-CSAT trigger queued. Reviewers also see a **Variance** action on each reviewed row — it compares the agent's self-evaluation against the reviewer score for the same call, per criterion.

## Statuses: pending → acknowledged → appealed → resolved

Every evaluation moves through a four-status lifecycle, and the actions available change at each step:

* **Pending** — a fresh score the agent hasn't reacted to. The agent can **Acknowledge** it or **Appeal** it with a written note. Pending rows that came from the auto-sampler or the low-CSAT trigger instead show a **Claim & score** action for reviewers, and an open, still-untouched human assignment shows **Reassign** to move it to a different evaluator.
* **Acknowledged** — the agent has accepted the score. An appeal is still possible while the status is pending or acknowledged.
* **Appealed** — the agent sent a written appeal note; reviewers see a **Resolve** action on the row.
* **Resolved** — the supervisor reviewed the appeal and closed it with an optional resolution note. Resolving does not regrade the score — it records that the appeal was reviewed.

Self-evaluations (the agent's **Evaluate my call**) land pre-acknowledged with a **Self** badge, are excluded from the official QA average, and skip the appeal loop entirely — there is no reviewer to appeal to on a self-score.

## AI auto-scoring with human override

The **Auto-scored** tab lists evaluations the AI auto-score pass wrote against your active form. A background sweep grades every completed conversation that has a transcript; rows scoring under your flag threshold (70 out of 100 by default) get a **flagged** badge so the supervisor's review queue carries only the borderline conversations. A **Flagged for review only** switch narrows the list to those rows.

If the tab shows "AI Auto-QA is off", enable it under **Settings → AI Auto-QA** — the sweep skips tenants that haven't opted in, so nothing is scored until you turn it on.

Each pending AI-scored row offers **Override**: a reviewer grades the same criteria and saves, and the server re-derives the weighted total from the form's weights — keeping an AI score correctable in place instead of creating a shadow copy.

## Self-evaluations and the variance view

An agent scores their own conversation with **Evaluate my call**: pick the scorecard and call id, grade yourself, and save. Self-evaluations are excluded from the official QA average by default — an agent can't inflate their own ranking by scoring themselves — they're for reflection, and they feed calibration.

Both reviewers and agents can open the **Variance** dialog on a row. It pairs the agent's self-evaluation with the reviewer's score for the same call and shows the total plus a per-criterion delta, so a coaching conversation starts from where the two readings diverge.

## Dashboards: trends and appeals

The **Workload** tab shows every eligible evaluator with their open-assignment count against the tenant's quota, remaining capacity, and how many of their open assignments are overdue — both values are always server-derived. From here you assign a specific conversation to a specific evaluator and set per-agent quotas and due dates. Reassigning an open assignment from the Evaluations tab rebalances without re-scoring.

For the org-wide trend view, the parent **Quality** dashboard charts per-day average scores across the official evaluations (reviewer-authored plus AI auto-scored), with breakdowns by agent and by scorecard form — the "is quality moving" read. **Calibration** on this page runs blind inter-rater sessions: a supervisor opens a session on one conversation, each reviewer scores without seeing the others' results, and the report compares each reviewer's delta from consensus — including how far the AI auto-score drifts from the human consensus.

Keep the **Appealed** status filter in view during the week: it isolates the queue of written agent appeals waiting on a resolution.

## How this differs from the LLM auto-score

Both pipelines write into the same evaluations list, and an agent can acknowledge or appeal either one. The difference is provenance: a manual evaluation carries the reviewer's identity, while an AI evaluation is flagged as auto-scored so downstream readers and your own percent-auto-scored reports can tell the two apart. Auto-scored conversations count toward official averages the same way human ones do, and the reviewer-override path corrects them in place.

If you're deciding between them: leave auto-scoring on for 100% coverage, and spend reviewer time spot-auditing the flagged rows plus a claimed share of sampled conversations rather than grading every conversation from scratch.

## API surface

Everything the page does is backed by the Quality API, so the same lifecycle runs from scripts and QA tooling. Reviewer-scope routes (owner / admin / supervisor) handle forms, scoring, overrides, and resolutions; agent-scope routes handle acknowledge, appeal, and self-evaluation.

| Operation                                                                     | Endpoint                                                                                  |     |                                          |
| ----------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | --- | ---------------------------------------- |
| Create / list / update evaluation forms                                       | \`POST                                                                                    | GET | PATCH /api/v1/quality/evaluation-forms\` |
| Score a conversation (reviewer)                                               | `POST /api/v1/quality/evaluations`                                                        |     |                                          |
| Self-evaluate your own call                                                   | `POST /api/v1/quality/evaluations/self`                                                   |     |                                          |
| List evaluations (status / agent / CSAT-triggered filters, keyset pagination) | `GET /api/v1/quality/evaluations`                                                         |     |                                          |
| Self-vs-reviewer variance for a call                                          | `GET /api/v1/quality/evaluations/variance`                                                |     |                                          |
| Acknowledge / appeal / resolve / claim-score / override                       | `POST /api/v1/quality/evaluations/{id}/{acknowledge,appeal,resolve,claim-score,override}` |     |                                          |

## See also

* [QA evaluations and the performance leaderboard](/concepts/qa-leaderboard-and-evaluations) — how graded scores roll up into the ranked leaderboard, and what the API test suite verifies about the excluded-from-official-averages rules
* [Call QA scorecards](/voice/call-qa-scorecards) — the call-level rubric on the call detail page, a fixed five-dimension grading sibling to the custom forms built here
* [QA sampling settings](/guides/qa-sampling-settings) — scheduled weekly auto-assignment of sampled conversations to evaluators
* [Searchable recording and transcript library](/voice/recording-library) — find the conversations a review needs by keyword instead of scrubbing audio