Build a contact-center QA program

This guide walks a supervisor through the whole Quality Management loop: you author a scorecard with weighted sections and a compliance gate, calibrate reviewers against each other before any score counts, grade calls and work the acknowledge/appeal lifecycle, then put agent effort on a leaderboard. Everything runs from the Quality section of the dashboard and the /api/v1/quality API. Working knowledge of the Quality Management API helps — this guide is the narrative; that page is the endpoint reference.

Who sees what

The Quality surfaces are role-scoped on both the dashboard and the API:

Owner / admin — full program: author forms, score, calibrate, set workloads, read the org rollup and the full leaderboard.
Supervisor — the same reviewer surface, scoped to the queues they supervise on the leaderboard (requesting a queue they do not supervise returns 403).
Agent — their own view only: the evaluations made about them (with acknowledge/appeal) and their own gamification card under Quality → Leaderboard.

1. Program shape

A working QA program has four moving parts, and each maps to one surface:

The rubric — an evaluation form: weighted sections, weighted criteria, and a sample_rate_pct that feeds the continuous auto-sampler (every 15 minutes; covers AI voice agents’ production calls as well as human agents’). Authored under Quality → Evaluations → Scorecards.
The reviewer queue — system-created pending rows (auto-sampled calls, AI auto-scored calls below your flag threshold) plus manual assignments land in Quality → Evaluations. Reviewers claim, correct, and resolve from there.
Reviewer agreement — blind calibration sessions under Quality → Evaluations → Calibration measure how far reviewers (and the AI auto-scorer) drift from each other before scores reach agents.
The agent-facing layer — the agent sees their own evaluations, acknowledges or appeals them, and watches their points and badges on Quality → Leaderboard, windowed by day, week, or month.

The hub page at Quality ties the surfaces together and also carries the cross-channel QM rollup (the org-wide scorecard across inbox and voice) plus the Practice Studio — AI-simulated-customer roleplay training where agents rehearse conversations and get auto-scored feedback. Practice is adjacent to QA scoring, not part of it: it never feeds an agent’s evaluation record or the leaderboard. It has its own guide: Train agents with AI roleplay in Practice Studio. The rest of this guide walks the loop in the order you would stand it up.

2. Author a form

Create the rubric under Quality → Evaluations → Scorecards, or over the API:

curl -X POST https://api.orbit.devotel.io/api/v1/quality/evaluation-forms \
  -H "X-API-Key: dv_live_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Inbound Support QA",
    "sample_rate_pct": 10,
    "definition": {
      "sections": [
        {
          "id": "compliance",
          "label": "Compliance",
          "weight": 1,
          "auto_fail": true,
          "criteria": [
            { "id": "identity-check", "label": "Verified identity before disclosure", "weight": 1 }
          ]
        },
        {
          "id": "soft-skills",
          "label": "Soft skills",
          "weight": 2,
          "criteria": [
            { "id": "empathy", "label": "Showed empathy", "weight": 1 },
            { "id": "clear-next-steps", "label": "Clear next steps", "weight": 1 }
          ]
        }
      ]
    }
  }'

Authoring decisions that matter later:

Weights are relative. Numbers normalize within each section and across the form — weight: 2 is simply twice weight: 1, whether you think in percentages or points.
auto_fail is the compliance gate. Mark a section auto_fail: true and any zero on one of its criteria forces the whole evaluation to 0. Use it for identity verification, required disclosures, and similar non-negotiables — not for soft skills.
sample_rate_pct decides coverage. 10 means the continuous auto-sampler (every 15 minutes) queues a weighted share of each agent’s calls against this form as pending rows. Leave it 0 to keep the form manual-only.
max_score per criterion defaults to 100; reviewers submit a raw 0..max_score value per criterion and the server derives the 0..100 total. A client can never set the total directly.

3. Run calibration before scores go live

Calibrate before you grade for real: with a brand-new form, reviewers will each read the rubric differently, and a calibration session surfaces that disagreement while nothing is at stake. Open Quality → Evaluations → Calibration and start a session on one call against the form you just authored, inviting the reviewers who will score against it. Each invited reviewer scores the call blind — nobody can see another reviewer’s score (or the AI’s) until they submit their own, and a reviewer who has not submitted cannot open the report at all. Over the API:

# 1. Create the session
curl -X POST https://api.orbit.devotel.io/api/v1/quality/calibrations \
  -H "X-API-Key: dv_live_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "call_id": "call_9f2c",
    "form_id": "form_qa_support",
    "reviewer_ids": ["user_evaluator_7", "user_evaluator_3"]
  }'

# 2. Each reviewer submits their own blind score (run once per reviewer)
curl -X POST https://api.orbit.devotel.io/api/v1/quality/calibrations/cal_abc123/scores \
  -H "X-API-Key: dv_live_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "scores": { "identity-check": 100, "empathy": 80, "clear-next-steps": 90 }
  }'

# 3. Read the agreement report once two or more reviewers have submitted
curl https://api.orbit.devotel.io/api/v1/quality/calibrations/cal_abc123/report \
  -H "X-API-Key: dv_live_sk_your_key_here"

The report computes per-criterion variance, overall agreement, each reviewer’s delta from consensus, and — if the call already has an AI auto-scored evaluation against the same form — the AI’s drift from the human consensus, seeded as a read-only baseline. High variance on a criterion means the rubric wording is ambiguous; fix the criterion label or add guidance, then calibrate again. Once the report shows tight agreement, close the session (POST /calibrations/{id}/close) and put the form into live rotation. Two properties make calibration safe to run liberally:

Each reviewer submits once — a second attempt returns 409, so a session is a single blind snapshot per reviewer.
Calibration rows are practice data. They never feed the agent’s evaluation record, the ack/appeal flow, or the leaderboard.

4. Score and work the lifecycle

Live scoring happens under Quality → Evaluations. Rows arrive three ways: the auto-sampler (un-scored stubs from sample_rate_pct), AI auto-scored rows (auto_scored: true, flagged below your tenant’s flag threshold), and manual assignments. A reviewer grades a call against the form by submitting the per-criterion scores map:

curl -X POST https://api.orbit.devotel.io/api/v1/quality/evaluations \
  -H "X-API-Key: dv_live_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "form_id": "form_qa_support",
    "agent_id": "user_agent_12",
    "call_id": "call_9f2c",
    "scores": { "identity-check": 100, "empathy": 80, "clear-next-steps": 90 }
  }'

From there the evaluation moves through a defined lifecycle, and each transition is role-checked:

# The agent acknowledges the evaluation (only the evaluated agent can)
curl -X POST https://api.orbit.devotel.io/api/v1/quality/evaluations/eval_abc123/acknowledge \
  -H "X-API-Key: dv_live_sk_your_key_here"

# ...or appeals it with a written note
curl -X POST https://api.orbit.devotel.io/api/v1/quality/evaluations/eval_abc123/appeal \
  -H "X-API-Key: dv_live_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{ "appeal_note": "Identity was verified via the account PIN before any disclosure." }'

# A reviewer closes the appeal
curl -X POST https://api.orbit.devotel.io/api/v1/quality/evaluations/eval_abc123/resolve \
  -H "X-API-Key: dv_live_sk_your_key_here"

The rules to internalize:

pending → acknowledged or pending → appealed → resolved. Only the evaluated agent can acknowledge or appeal; an appeal requires an appeal_note; only a reviewer can resolve. Resolving closes the appeal — it does not regrade the score.
Auto-sampled rows are claimed, not edited. An un-scored sampler row is filled in with POST /evaluations/{id}/claim-score; only a pending, still-unclaimed auto-sampled row is eligible.
AI auto-scored rows are overridden, not rescored from scratch. A reviewer corrects one with POST /evaluations/{id}/override; the corrected total is re-derived, the row is stamped with the reviewer, and flagged_for_review clears.
Listing is keyset-paginated — read next_cursor and pass it back as cursor. Agents listing evaluations only ever see their own rows.

5. Workload and quotas

Once scoring is running, reviews need an owner or they sit unclaimed. Workload management — assigning a specific call to a specific evaluator, capping how many open reviews each evaluator holds, and tracking due dates — is covered in its own guide: QA workload management. Use it to turn the shared queue into named responsibility per evaluator.

6. Gamification leaderboard

Quality → Leaderboard ranks agents on points and badges computed on demand from the QA, call, and CSAT data the platform already captures — no extra setup, no separate ingestion. Window the board with Today / This week / This month, or narrow it to a single voice queue for a team competition. Over the API:

curl -X POST https://api.orbit.devotel.io/api/v1/quality/gamification/leaderboard \
  -H "X-API-Key: dv_live_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{ "period": "week", "queue_id": "queue_support_tier1" }'

Scoping is role-checked: owners and admins see the full roster; a supervisor sees only the queues they supervise and gets 403 for anything else. An agent reading Quality → Leaderboard sees their own card (GET /quality/gamification/me) — points, badges, and rank — computed against the same default board, so an agent’s rank always matches what their supervisor sees. Before adopting a new rewards scheme, pass point_rules or badge_definitions in the leaderboard body to preview a tuned ruleset against live data, and compare it against the defaults from GET /quality/gamification/config.

End-to-end sequence

Putting it together, a minimal program in API calls:

POST /evaluation-forms — author the rubric with weights, one auto_fail section, and a starting sample_rate_pct.
POST /calibrations → each reviewer POST /calibrations/{id}/scores → GET /calibrations/{id}/report → POST /calibrations/{id}/close — calibrate until variance is small.
Reviewers grade live work: POST /evaluations for manual reviews; POST /evaluations/{id}/claim-score for sampler rows; POST /evaluations/{id}/override for flagged AI rows.
The agent reacts: POST /evaluations/{id}/acknowledge, or POST /evaluations/{id}/appeal → reviewer POST /evaluations/{id}/resolve.
POST /gamification/leaderboard — read the week’s board; the agent checks their own card with GET /gamification/me.

WhatsApp

Guides

Messaging Controls

Audience & Support Tools

Build a contact-center QA program

Build a contact-center QA program

Who sees what

1. Program shape

2. Author a form

3. Run calibration before scores go live

4. Score and work the lifecycle

5. Workload and quotas

6. Gamification leaderboard

End-to-end sequence

See also

​Build a contact-center QA program

​Who sees what

​1. Program shape

​2. Author a form

​3. Run calibration before scores go live

​4. Score and work the lifecycle

​5. Workload and quotas

​6. Gamification leaderboard

​End-to-end sequence

​See also

Build a contact-center QA program

Who sees what

1. Program shape

2. Author a form

3. Run calibration before scores go live

4. Score and work the lifecycle

5. Workload and quotas

6. Gamification leaderboard

End-to-end sequence

See also