Build a contact-center QA program

This guide walks a supervisor through the whole Quality Management loop: you author a scorecard with weighted sections and a compliance gate, calibrate reviewers against each other before any score counts, grade calls and work the acknowledge/appeal lifecycle, then put agent effort on a leaderboard. Everything runs from the Quality section of the dashboard and the /api/v1/quality API. Working knowledge of the Quality Management API helps — this guide is the narrative; that page is the endpoint reference.

Who sees what

The Quality surfaces are role-scoped on both the dashboard and the API:

Owner / admin — full program: author forms, score, calibrate, set workloads, read the org rollup and the full leaderboard.
Supervisor — the same reviewer surface, scoped to the queues they supervise on the leaderboard (requesting a queue they do not supervise returns 403).
Agent — their own view only: the evaluations made about them (with acknowledge/appeal) and their own gamification card under Quality → Leaderboard.

1. Program shape

A working QA program has four moving parts, and each maps to one surface:

The rubric — an evaluation form: weighted sections, weighted criteria, and a sample_rate_pct that feeds the continuous auto-sampler (every 15 minutes; covers AI voice agents’ production calls as well as human agents’). Authored under Quality → Evaluations → Scorecards.
The reviewer queue — system-created pending rows (auto-sampled calls, AI auto-scored calls below your flag threshold) plus manual assignments land in Quality → Evaluations. Reviewers claim, correct, and resolve from there.
Reviewer agreement — blind calibration sessions under Quality → Evaluations → Calibration measure how far reviewers (and the AI auto-scorer) drift from each other before scores reach agents.
The agent-facing layer — the agent sees their own evaluations, acknowledges or appeals them, and watches their points and badges on Quality → Leaderboard, windowed by day, week, or month.

The hub page at Quality ties the surfaces together and also carries the cross-channel QM rollup (the org-wide scorecard across inbox and voice) plus the Practice Studio — AI-simulated-customer roleplay training where agents rehearse conversations and get auto-scored feedback. Practice is adjacent to QA scoring, not part of it: it never feeds an agent’s evaluation record or the leaderboard. It has its own guide: Train agents with AI roleplay in Practice Studio. The rest of this guide walks the loop in the order you would stand it up.

2. Author a form

Create the rubric under Quality → Evaluations → Scorecards, or over the API:

curl -X POST https://api.orbit.devotel.io/api/v1/quality/evaluation-forms \
  -H "X-API-Key: dv_live_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Inbound Support QA",
    "sample_rate_pct": 10,
    "definition": {
      "sections": [
        {
          "id": "compliance",
          "label": "Compliance",
          "weight": 1,
          "auto_fail": true,
          "criteria": [
            { "id": "identity-check", "label": "Verified identity before disclosure", "weight": 1 }
          ]
        },
        {
          "id": "soft-skills",
          "label": "Soft skills",
          "weight": 2,
          "criteria": [
            { "id": "empathy", "label": "Showed empathy", "weight": 1 },
            { "id": "clear-next-steps", "label": "Clear next steps", "weight": 1 }
          ]
        }
      ]
    }
  }'

Authoring decisions that matter later:

Weights are relative. Numbers normalize within each section and across the form — weight: 2 is simply twice weight: 1, whether you think in percentages or points.
auto_fail is the compliance gate. Mark a section auto_fail: true and any zero on one of its criteria forces the whole evaluation to 0. Use it for identity verification, required disclosures, and similar non-negotiables — not for soft skills.
sample_rate_pct decides coverage. 10 means the continuous auto-sampler (every 15 minutes) queues a weighted share of each agent’s calls against this form as pending rows. Leave it 0 to keep the form manual-only.
max_score per criterion defaults to 100; reviewers submit a raw 0..max_score value per criterion and the server derives the 0..100 total. A client can never set the total directly.

3. Run calibration before scores go live

Calibrate before you grade for real: with a brand-new form, reviewers will each read the rubric differently, and a calibration session surfaces that disagreement while nothing is at stake. Open Quality → Evaluations → Calibration and start a session on one call against the form you just authored, inviting the reviewers who will score against it. Each invited reviewer scores the call blind — nobody can see another reviewer’s score (or the AI’s) until they submit their own, and a reviewer who has not submitted cannot open the report at all. Over the API:

# 1. Create the session
curl -X POST https://api.orbit.devotel.io/api/v1/quality/calibrations \
  -H "X-API-Key: dv_live_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "call_id": "call_9f2c",
    "form_id": "form_qa_support",
    "reviewer_ids": ["user_evaluator_7", "user_evaluator_3"]
  }'

# 2. Each reviewer submits their own blind score (run once per reviewer)
curl -X POST https://api.orbit.devotel.io/api/v1/quality/calibrations/cal_abc123/scores \
  -H "X-API-Key: dv_live_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "scores": { "identity-check": 100, "empathy": 80, "clear-next-steps": 90 }
  }'

# 3. Read the agreement report once two or more reviewers have submitted
curl https://api.orbit.devotel.io/api/v1/quality/calibrations/cal_abc123/report \
  -H "X-API-Key: dv_live_sk_your_key_here"

The report computes per-criterion variance, overall agreement, each reviewer’s delta from consensus, and — if the call already has an AI auto-scored evaluation against the same form — the AI’s drift from the human consensus, seeded as a read-only baseline. High variance on a criterion means the rubric wording is ambiguous; fix the criterion label or add guidance, then calibrate again. Once the report shows tight agreement, close the session (POST /calibrations/{id}/close) and put the form into live rotation. Two properties make calibration safe to run liberally:

Each reviewer submits once — a second attempt returns 409, so a session is a single blind snapshot per reviewer.
Calibration rows are practice data. They never feed the agent’s evaluation record, the ack/appeal flow, or the leaderboard.

4. Score and work the lifecycle

Live scoring happens under Quality → Evaluations. Rows arrive three ways: the auto-sampler (un-scored stubs from sample_rate_pct), AI auto-scored rows (auto_scored: true, flagged below your tenant’s flag threshold), and manual assignments. A reviewer grades a call against the form by submitting the per-criterion scores map:

curl -X POST https://api.orbit.devotel.io/api/v1/quality/evaluations \
  -H "X-API-Key: dv_live_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "form_id": "form_qa_support",
    "agent_id": "user_agent_12",
    "call_id": "call_9f2c",
    "scores": { "identity-check": 100, "empathy": 80, "clear-next-steps": 90 }
  }'

From there the evaluation moves through a defined lifecycle, and each transition is role-checked:

# The agent acknowledges the evaluation (only the evaluated agent can)
curl -X POST https://api.orbit.devotel.io/api/v1/quality/evaluations/eval_abc123/acknowledge \
  -H "X-API-Key: dv_live_sk_your_key_here"

# ...or appeals it with a written note
curl -X POST https://api.orbit.devotel.io/api/v1/quality/evaluations/eval_abc123/appeal \
  -H "X-API-Key: dv_live_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{ "appeal_note": "Identity was verified via the account PIN before any disclosure." }'

# A reviewer closes the appeal
curl -X POST https://api.orbit.devotel.io/api/v1/quality/evaluations/eval_abc123/resolve \
  -H "X-API-Key: dv_live_sk_your_key_here"

The rules to internalize:

pending → acknowledged or pending → appealed → resolved. Only the evaluated agent can acknowledge or appeal; an appeal requires an appeal_note; only a reviewer can resolve. Resolving closes the appeal — it does not regrade the score.
Auto-sampled rows are claimed, not edited. An un-scored sampler row is filled in with POST /evaluations/{id}/claim-score; only a pending, still-unclaimed auto-sampled row is eligible.
AI auto-scored rows are overridden, not rescored from scratch. A reviewer corrects one with POST /evaluations/{id}/override; the corrected total is re-derived, the row is stamped with the reviewer, and flagged_for_review clears.
Listing is keyset-paginated — read next_cursor and pass it back as cursor. Agents listing evaluations only ever see their own rows.

5. The AI call-quality coach: auto-score → coaching plan

Once scoring is live, the QA loop should not stop at the score. The auto-coaching loop closes the last step automatically: when a score crosses a configured threshold, the platform writes a coaching_plans row for the handling agent without waiting for a supervisor to notice — so a chronically poor pattern turns into an actionable coaching assignment the moment it shows up, not after a quarterly review. Three trigger sources close the loop:

A QA score below your coaching threshold. Every human-authored evaluation finalization (create, claim-score, or an AI-score override) is checked against the coaching threshold. A score strictly below the threshold (default 70/100, equal to the Five9/NICE QM passing bar) auto-assigns a coaching plan for the evaluated agent on the same write. Self-evaluations never trigger — a reviewer scoring their own call cannot spin up their own coaching plan.
A detractor CSAT response. A very low survey answer (default bottom-band only, 1/5, stricter than the QA-review trigger) fires a coaching plan against the assigned handling agent, separate from the QA-review queue the same trigger already feeds.
A sustained intraday adherence breach. The intraday WFM scheduler watches adherence ratios and assigns a coaching plan when adherence stays strictly below the default 0.7 ratio — reserved for chronic, material slips rather than one-off schedule drift.

Every auto-assigned plan carries a per-source tag on its goal ([auto-coaching:qa_score] / [auto-coaching:csat] / [auto-coaching:adherence]) plus the system:auto-coaching sentinel owner, so you can tell a machine-assigned plan apart from a supervisor-authored one in the normal Coaching Plans surface. Idempotency is source-specific: at most one OPEN auto-assigned plan exists per agent per trigger source, so a chronically low-scoring agent accumulates a single visible plan per signal rather than a fresh plan every time they trip the threshold. Tuning and visibility:

Thresholds are operator-tuned via the COACHING_QA_SCORE_TRIGGER_THRESHOLD, COACHING_CSAT_TRIGGER_THRESHOLD, and COACHING_ADHERENCE_TRIGGER_THRESHOLD environment variables — there is deliberately no per-tenant UI authoring surface. Reading the effective configuration with preview: GET /api/v1/quality/coaching-rules (reviewer scope — owner / admin / supervisor) reflects the live values.
Compliance flags flow into the same loop. An AI auto-scored evaluation marked flagged_for_review below your qa_autoscore.flag_threshold appears in the QA Evaluations needs-human-review queue, and a supervisor’s corrective override is itself a human finalization — so it also passes through isLowQaScoreForCoaching and can assign a coaching plan without any extra wiring on your side.
Coaching cards appear beside the scorecard. The per-agent view at Quality → Agents → your-agent carries the QA score, compliance-flag count, CSAT, and WFM adherence strip for the window, plus the recent-evaluated-interactions list — the tie-in that makes “the AI scored this below the flag threshold, auto-coaching assigned” a one-glance answer when a coaching plan lands.

A coaching plan assigned by an AI score is handled exactly like a supervisor-authored one once created: the supervisor sees it under Voice → Coaching → Coaching plans, can attach training modules to it, and closes it out when the agent recovers. Nothing in the auto-assignments surface is a second workflow to learn.

6. Workload and quotas

Once scoring is running, reviews need an owner or they sit unclaimed. Workload management — assigning a specific call to a specific evaluator, capping how many open reviews each evaluator holds, and tracking due dates — is covered in its own guide: QA workload management. Use it to turn the shared queue into named responsibility per evaluator.

7. Gamification leaderboard

Quality → Leaderboard ranks agents on points and badges computed on demand from the QA, call, and CSAT data the platform already captures — no extra setup, no separate ingestion. Window the board with Today / This week / This month, or narrow it to a single voice queue for a team competition. Over the API:

curl -X POST https://api.orbit.devotel.io/api/v1/quality/gamification/leaderboard \
  -H "X-API-Key: dv_live_sk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{ "period": "week", "queue_id": "queue_support_tier1" }'

Scoping is role-checked: owners and admins see the full roster; a supervisor sees only the queues they supervise and gets 403 for anything else. An agent reading Quality → Leaderboard sees their own card (GET /quality/gamification/me) — points, badges, and rank — computed against the same default board, so an agent’s rank always matches what their supervisor sees. Before adopting a new rewards scheme, pass point_rules or badge_definitions in the leaderboard body to preview a tuned ruleset against live data, and compare it against the defaults from GET /quality/gamification/config.

End-to-end sequence

Putting it together, a minimal program in API calls:

POST /evaluation-forms — author the rubric with weights, one auto_fail section, and a starting sample_rate_pct.
POST /calibrations → each reviewer POST /calibrations/{id}/scores → GET /calibrations/{id}/report → POST /calibrations/{id}/close — calibrate until variance is small.
Reviewers grade live work: POST /evaluations for manual reviews; POST /evaluations/{id}/claim-score for sampler rows; POST /evaluations/{id}/override for flagged AI rows.
The agent reacts: POST /evaluations/{id}/acknowledge, or POST /evaluations/{id}/appeal → reviewer POST /evaluations/{id}/resolve.
When a score crosses a coaching threshold, the platform auto-assigns a coaching plan — read the effective thresholds with GET /quality/coaching-rules and close the resulting plan under Voice → Coaching like any supervisor-authored one.
POST /gamification/leaderboard — read the week’s board; the agent checks their own card with GET /gamification/me.

WhatsApp

Developer console

Guides

Messaging Controls

Audience & Support Tools

Build a contact-center QA program

Build a contact-center QA program

Who sees what

1. Program shape

2. Author a form

3. Run calibration before scores go live

4. Score and work the lifecycle

5. The AI call-quality coach: auto-score → coaching plan

6. Workload and quotas

7. Gamification leaderboard

End-to-end sequence

See also

​Build a contact-center QA program

​Who sees what

​1. Program shape

​2. Author a form

​3. Run calibration before scores go live

​4. Score and work the lifecycle

​5. The AI call-quality coach: auto-score → coaching plan

​6. Workload and quotas

​7. Gamification leaderboard

​End-to-end sequence

​See also

Build a contact-center QA program

Who sees what

1. Program shape

2. Author a form

3. Run calibration before scores go live

4. Score and work the lifecycle

5. The AI call-quality coach: auto-score → coaching plan

6. Workload and quotas

7. Gamification leaderboard

End-to-end sequence

See also