> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Build a contact-center QA program

> Set up a full quality program: author a weighted scorecard, run blind calibration so reviewers agree, score calls through the ack/appeal lifecycle, and read the gamification leaderboard.

# Build a contact-center QA program

This guide walks a supervisor through the whole Quality Management loop: you author a scorecard with weighted sections and a compliance gate, calibrate reviewers against each other before any score counts, grade calls and work the acknowledge/appeal lifecycle, then put agent effort on a leaderboard. Everything runs from the **Quality** section of the dashboard and the `/api/v1/quality` API.

Working knowledge of the [Quality Management API](/api-reference/quality) helps — this guide is the narrative; that page is the endpoint reference.

## Who sees what

The Quality surfaces are role-scoped on both the dashboard and the API:

* **Owner / admin** — full program: author forms, score, calibrate, set workloads, read the org rollup and the full leaderboard.
* **Supervisor** — the same reviewer surface, scoped to the queues they supervise on the leaderboard (requesting a queue they do not supervise returns `403`).
* **Agent** — their own view only: the evaluations made about them (with acknowledge/appeal) and their own gamification card under **Quality → Leaderboard**.

## 1. Program shape

A working QA program has four moving parts, and each maps to one surface:

1. **The rubric** — an evaluation form: weighted sections, weighted criteria, and a `sample_rate_pct` that feeds the continuous auto-sampler (every 15 minutes; covers AI voice agents' production calls as well as human agents'). Authored under **Quality → Evaluations → Scorecards**.
2. **The reviewer queue** — system-created pending rows (auto-sampled calls, AI auto-scored calls below your flag threshold) plus manual assignments land in **Quality → Evaluations**. Reviewers claim, correct, and resolve from there.
3. **Reviewer agreement** — blind calibration sessions under **Quality → Evaluations → Calibration** measure how far reviewers (and the AI auto-scorer) drift from each other *before* scores reach agents.
4. **The agent-facing layer** — the agent sees their own evaluations, acknowledges or appeals them, and watches their points and badges on **Quality → Leaderboard**, windowed by day, week, or month.

The hub page at **Quality** ties the surfaces together and also carries the cross-channel QM rollup (the org-wide scorecard across inbox and voice) plus the **Practice Studio** — AI-simulated-customer roleplay training where agents rehearse conversations and get auto-scored feedback. Practice is adjacent to QA scoring, not part of it: it never feeds an agent's evaluation record or the leaderboard. It has its own guide: [Train agents with AI roleplay in Practice Studio](/guides/practice-studio-roleplay).

The rest of this guide walks the loop in the order you would stand it up.

## 2. Author a form

Create the rubric under **Quality → Evaluations → Scorecards**, or over the API:

```bash theme={null}
curl -X POST https://api.orbit.devotel.io/api/v1/quality/evaluation-forms \
  -H "X-API-Key: dv_live_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Inbound Support QA",
    "sample_rate_pct": 10,
    "definition": {
      "sections": [
        {
          "id": "compliance",
          "label": "Compliance",
          "weight": 1,
          "auto_fail": true,
          "criteria": [
            { "id": "identity-check", "label": "Verified identity before disclosure", "weight": 1 }
          ]
        },
        {
          "id": "soft-skills",
          "label": "Soft skills",
          "weight": 2,
          "criteria": [
            { "id": "empathy", "label": "Showed empathy", "weight": 1 },
            { "id": "clear-next-steps", "label": "Clear next steps", "weight": 1 }
          ]
        }
      ]
    }
  }'
```

Authoring decisions that matter later:

* **Weights are relative.** Numbers normalize within each section and across the form — `weight: 2` is simply twice `weight: 1`, whether you think in percentages or points.
* **`auto_fail` is the compliance gate.** Mark a section `auto_fail: true` and any zero on one of its criteria forces the whole evaluation to `0`. Use it for identity verification, required disclosures, and similar non-negotiables — not for soft skills.
* **`sample_rate_pct` decides coverage.** `10` means the continuous auto-sampler (every 15 minutes) queues a weighted share of each agent's calls against this form as pending rows. Leave it `0` to keep the form manual-only.
* **`max_score` per criterion** defaults to `100`; reviewers submit a raw `0..max_score` value per criterion and the server derives the `0..100` total. A client can never set the total directly.

## 3. Run calibration before scores go live

Calibrate before you grade for real: with a brand-new form, reviewers will each read the rubric differently, and a calibration session surfaces that disagreement while nothing is at stake.

Open **Quality → Evaluations → Calibration** and start a session on one call against the form you just authored, inviting the reviewers who will score against it. Each invited reviewer scores the call **blind** — nobody can see another reviewer's score (or the AI's) until they submit their own, and a reviewer who has not submitted cannot open the report at all. Over the API:

```bash theme={null}
# 1. Create the session
curl -X POST https://api.orbit.devotel.io/api/v1/quality/calibrations \
  -H "X-API-Key: dv_live_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "call_id": "call_9f2c",
    "form_id": "form_qa_support",
    "reviewer_ids": ["user_evaluator_7", "user_evaluator_3"]
  }'

# 2. Each reviewer submits their own blind score (run once per reviewer)
curl -X POST https://api.orbit.devotel.io/api/v1/quality/calibrations/cal_abc123/scores \
  -H "X-API-Key: dv_live_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "scores": { "identity-check": 100, "empathy": 80, "clear-next-steps": 90 }
  }'

# 3. Read the agreement report once two or more reviewers have submitted
curl https://api.orbit.devotel.io/api/v1/quality/calibrations/cal_abc123/report \
  -H "X-API-Key: dv_live_sk_your_key_here"
```

The report computes per-criterion variance, overall agreement, each reviewer's delta from consensus, and — if the call already has an AI auto-scored evaluation against the same form — the AI's drift from the human consensus, seeded as a read-only baseline. High variance on a criterion means the rubric wording is ambiguous; fix the criterion label or add guidance, then calibrate again. Once the report shows tight agreement, close the session (`POST /calibrations/{id}/close`) and put the form into live rotation.

Two properties make calibration safe to run liberally:

* **Each reviewer submits once** — a second attempt returns `409`, so a session is a single blind snapshot per reviewer.
* **Calibration rows are practice data.** They never feed the agent's evaluation record, the ack/appeal flow, or the leaderboard.

## 4. Score and work the lifecycle

Live scoring happens under **Quality → Evaluations**. Rows arrive three ways: the auto-sampler (un-scored stubs from `sample_rate_pct`), AI auto-scored rows (`auto_scored: true`, flagged below your tenant's flag threshold), and manual assignments. A reviewer grades a call against the form by submitting the per-criterion scores map:

```bash theme={null}
curl -X POST https://api.orbit.devotel.io/api/v1/quality/evaluations \
  -H "X-API-Key: dv_live_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "form_id": "form_qa_support",
    "agent_id": "user_agent_12",
    "call_id": "call_9f2c",
    "scores": { "identity-check": 100, "empathy": 80, "clear-next-steps": 90 }
  }'
```

From there the evaluation moves through a defined lifecycle, and each transition is role-checked:

```bash theme={null}
# The agent acknowledges the evaluation (only the evaluated agent can)
curl -X POST https://api.orbit.devotel.io/api/v1/quality/evaluations/eval_abc123/acknowledge \
  -H "X-API-Key: dv_live_sk_your_key_here"

# ...or appeals it with a written note
curl -X POST https://api.orbit.devotel.io/api/v1/quality/evaluations/eval_abc123/appeal \
  -H "X-API-Key: dv_live_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{ "appeal_note": "Identity was verified via the account PIN before any disclosure." }'

# A reviewer closes the appeal
curl -X POST https://api.orbit.devotel.io/api/v1/quality/evaluations/eval_abc123/resolve \
  -H "X-API-Key: dv_live_sk_your_key_here"
```

The rules to internalize:

* **`pending → acknowledged`** or **`pending → appealed → resolved`.** Only the evaluated agent can acknowledge or appeal; an appeal requires an `appeal_note`; only a reviewer can resolve. Resolving closes the appeal — it does not regrade the score.
* **Auto-sampled rows are claimed, not edited.** An un-scored sampler row is filled in with `POST /evaluations/{id}/claim-score`; only a `pending`, still-unclaimed auto-sampled row is eligible.
* **AI auto-scored rows are overridden, not rescored from scratch.** A reviewer corrects one with `POST /evaluations/{id}/override`; the corrected total is re-derived, the row is stamped with the reviewer, and `flagged_for_review` clears.
* **Listing is keyset-paginated** — read `next_cursor` and pass it back as `cursor`. Agents listing evaluations only ever see their own rows.

## 5. Workload and quotas

Once scoring is running, reviews need an owner or they sit unclaimed. Workload management — assigning a specific call to a specific evaluator, capping how many open reviews each evaluator holds, and tracking due dates — is covered in its own guide: [QA workload management](/guides/qa-workload-management). Use it to turn the shared queue into named responsibility per evaluator.

## 6. Gamification leaderboard

**Quality → Leaderboard** ranks agents on points and badges computed on demand from the QA, call, and CSAT data the platform already captures — no extra setup, no separate ingestion. Window the board with **Today / This week / This month**, or narrow it to a single voice queue for a team competition. Over the API:

```bash theme={null}
curl -X POST https://api.orbit.devotel.io/api/v1/quality/gamification/leaderboard \
  -H "X-API-Key: dv_live_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{ "period": "week", "queue_id": "queue_support_tier1" }'
```

Scoping is role-checked: owners and admins see the full roster; a supervisor sees only the queues they supervise and gets `403` for anything else. An agent reading **Quality → Leaderboard** sees their own card (`GET /quality/gamification/me`) — points, badges, and rank — computed against the same default board, so an agent's rank always matches what their supervisor sees.

Before adopting a new rewards scheme, pass `point_rules` or `badge_definitions` in the leaderboard body to preview a tuned ruleset against live data, and compare it against the defaults from `GET /quality/gamification/config`.

## End-to-end sequence

Putting it together, a minimal program in API calls:

1. `POST /evaluation-forms` — author the rubric with weights, one `auto_fail` section, and a starting `sample_rate_pct`.
2. `POST /calibrations` → each reviewer `POST /calibrations/{id}/scores` → `GET /calibrations/{id}/report` → `POST /calibrations/{id}/close` — calibrate until variance is small.
3. Reviewers grade live work: `POST /evaluations` for manual reviews; `POST /evaluations/{id}/claim-score` for sampler rows; `POST /evaluations/{id}/override` for flagged AI rows.
4. The agent reacts: `POST /evaluations/{id}/acknowledge`, or `POST /evaluations/{id}/appeal` → reviewer `POST /evaluations/{id}/resolve`.
5. `POST /gamification/leaderboard` — read the week's board; the agent checks their own card with `GET /gamification/me`.

## See also

* [Quality Management API](/api-reference/quality) — full endpoint reference for every call above
* [QA workload management](/guides/qa-workload-management) — assignments, quotas, and due dates
* [Train agents with AI roleplay in Practice Studio](/guides/practice-studio-roleplay) — the before-call side: agents rehearse against an AI customer
* [Error codes](/api-reference/error-codes) — `409` state conflicts and `422 VALIDATION_ERROR` shapes