The call-quality evaluation lifecycle

Quality management has two halves. The first is the scorecard pipeline — evaluation forms, reviewer scorecards, acknowledge/appeal, AI auto-scoring, and the leaderboard — covered in QA evaluations and the performance leaderboard. The second half, covered here, is the operations layer around those scorecards: routing review work to humans, keeping reviewers consistent with each other, turning QA output into coaching, verifying that auto-scoring is actually covering your traffic, and rolling per-call scores up into org-wide summary, trends, drivers, and custom supervisor composites. The ownership boundary in one line: this page owns routing and calibration; the sibling page owns the scorecards themselves. Forms, weighted criteria, the author-acknowledge-appeal-resolve lifecycle, and leaderboard points are all documented there. Once a score exists, everything below — who was asked to produce it, how reviewers stay honest, how the operation is measured — lives here. A call finishes, a recording and transcript land in the library, and then one of two things happens: the AI auto-scores it, or a supervisor assigns it to a human evaluator as a scoring task. Everything below is what happens to that scoring task — from assignment through calibration to the rollups supervisors watch.

The cast of entities

Evaluation assignment — a routing envelope for a scoring task, not a score. Assigning picks a call and names a reviewer; the row later carries the reviewer’s scorecard. An assignment stays open until the assigned reviewer scores the call, so open-assignment count is the workload measure. Supervisors rebalance open work between reviewers, every reviewer holds a quota of open assignments (the API rejects over-quota assigns with 409 QUOTA_EXCEEDED), and a call that already has an open assignment cannot be double-assigned (409 ALREADY_ASSIGNED). Due dates are derived from your tenant’s assignment window (created_at + due_days), not stamped on the row — change the window in settings and the workload view recomputes.
Calibration session — a scheduled inter-rater agreement pass that pins one call and one form and invites a roster of reviewers. Every invited reviewer scores the call blind: they cannot read another reviewer’s score — or the AI auto-score — until they submit their own, and that rule is enforced server-side, not just hidden in the UI. Once at least two reviewers submit, the session report shows per-criterion variance, overall agreement, and each reviewer’s delta from group consensus — including the AI’s drift from that consensus. Calibration scores are practice rows: they never enter official averages, the leaderboard, or the agent’s acknowledge/appeal flow.
Coaching cards — per-agent readouts folded from official reviewer-authored evaluations: average QA score, flagged share, weakest criteria, plus short badge chips. Supervisors read them as a “who needs drilling” view; agents read their own card scoped to themselves.
Autoscore coverage — the one-glance answer to “how much of my traffic is the automated AI scorecard actually scoring?” It counts completed calls in the trailing window against calls carrying an AI-scored evaluation, across both the voice scorecard lane and the conversation-judge lane, and averages the automated scores.
Auto-QA readiness — the operational health snapshot for the automation: is the auto-scoring toggle on, does an active evaluation form exist, are eligible calls landing in the window, and what share of those candidates actually came back with an AI scorecard. Coverage tells you how much; readiness tells you why not when the number is wrong — a disabled toggle, no active form, or calls without transcripts.

Two design decisions shape everything downstream, and both exist to protect the score that comes out at the end. Open-assignment quotas keep scoring honest. A reviewer with an unbounded queue of open work does one of two things: they rush to drain it and scores get thinner, or it ages and the operation stops meaning anything. A per-reviewer cap (rejected at the API with 409 QUOTA_EXCEEDED) forces backlog to stay visible as a supervisor problem — rebalance, reassign, or adjust the sample rate — instead of being silently absorbed into someone’s queue. And the queue only counts open work: a scored assignment drops out of the workload immediately, so the number a supervisor sees is the work actually left, not the work ever handed out. The blind rule keeps calibration signal-clean. Calibration measures whether two reviewers — or a reviewer and the AI — grade the same call the same way. That measurement is only valid if each score is formed independently. If reviewer B could read reviewer A’s submitted score before writing their own, agreement would inflate: B grades toward A, the report shows consensus, and the drift you needed to find has been laundered out. The API therefore gates the report reads — the detailed scores table and pairwise deltas return 403 until every invited reviewer who has not yet submitted their own score is asking for someone else’s. Submit yours first; then the report opens. That is why the blind rule lives server-side: a UI-hidden table would still be one network call away from an inflated agreement pass.

The five-step lifecycle

The operation below is one loop a supervisor runs continuously. Each step names the supervisor action that advances it, and the read endpoint that observes it.

Open. A supervisor opens an assignment — hands a specific call to a specific reviewer — or samples work automatically: a form with a sample_rate_pct above zero queues a share of each agent’s calls for human review, and with auto-scoring enabled every completed call is AI-scored in the background. Assignments respect per-reviewer quotas, and an open assignment can be reassigned when workloads shift. Advancing action: POST /api/v1/quality/evaluations/assign (or a reassign). Observed by: GET /api/v1/quality/evaluations/workload — per-reviewer open counts against their quota.
Calibrate. Calibration sessions sit alongside production scoring as a consistency pass — reviewers score the same call blind, and the session report exposes drift per reviewer, per criterion, and for the AI against group consensus. A team whose reviewers disagree on the calibration call will disagree on live ones; fix the rubric or the coaching, then re-run. Advancing action: POST /api/v1/quality/calibrations then POST /api/v1/quality/calibrations/{id}/scores per reviewer. Observed by: GET /api/v1/quality/calibrations/{id}/report, once each reviewer has submitted.
Score and react. The assigned reviewer scores the call; the agent acknowledges or appeals; a supervisor resolves. Low scores auto-generate coaching assignments, and coaching cards accumulate the per-agent picture. Advancing action: POST /api/v1/quality/evaluations (or claim-score on an auto-queued sample), then the acknowledge/appeal/resolve endpoints — the full sequence is documented in QA evaluations and the performance leaderboard. Observed by: GET /api/v1/quality/coaching-cards and GET /api/v1/quality/compliance-flags.
Roll up. The supervisor-level summary folds both scoring pipelines — reviewer-authored call scorecards and the AI-judged conversation scores on digital channels — into one org-wide view: pass rate, average confidence, breakdowns by channel and rubric, plus a recent-failures triage list. Trends give the per-day weighted-score average per agent and per form. Drivers rank the criteria. The gamification leaderboard ranks agents on the tuned points mix. Supervisors who want one number build a composite scorecard over the same facts. Advancing action: none — this step is pure read over graded facts. Observed by: GET /api/v1/quality/summary, GET /api/v1/quality/trends, GET /api/v1/quality/drivers, plus POST /api/v1/quality/scorecards/composite for the one-number view.
Verify coverage. Autoscore coverage and auto-QA readiness close the loop on automation itself: what share of traffic the AI scored, what it averaged, and — when coverage sags — which generation gate is off. Advancing action: none — these are health reads on the automation. Observed by: GET /api/v1/quality/autoscore-coverage and GET /api/v1/quality/auto-qa-readiness.

Steps 1 and 2 are the only ones with a human gate: someone assigns, and the reviewers submit. Everything after step 3 is a read over the graded facts the first three steps produced — which is why the API surface is so heavily read-weighted. The worked samples below run each of those steps over the wire. Every request and response shape mirrors the shipped handlers exactly — the request body is the Zod schema the route parses, the data.data envelope is what createResponse wraps, and the error envelope is what a rejected call returns. The full field-by-field contract for all twelve endpoints stays in the Quality API reference; these samples show how the calls chain together, not a second schema source.

Sampling via API

Run the sampling loop end-to-end: raise the form’s sampling share, assign one call from the sampled queue, hit the quota, and resolve it. Sampling itself is a form property — set sample_rate_pct above 0 on the form and the auto-sampler queues that share of each agent’s completed calls for human review. When you want a specific call scored directly, assign it by hand and observe the result on the workload read.

Node.js
Python
Go

// Orbit Node SDK — generic escape hatch (the typed helpers stop
// short of the quality/ namespace); reads ORBIT_API_KEY from env.
import { Orbit } from "@devotel/sdk-node";

const client = Orbit.fromEnv();

// 1. Turn on sampling: 10% of each agent's calls get queued for review.
await client.request("PATCH", "/quality/evaluation-forms/form_9j2qkx", {
  sample_rate_pct: 10,
});

// 2. Hand one specific call to a reviewer.
const assigned = await client.request("POST", "/quality/evaluations/assign", {
  form_id: "form_9j2qkx",
  call_id: "call_8h2xq7",
  agent_id: "agent_2k9xd4",
  assignee_id: "user_2f8Kq1XvB",
});
console.log(assigned.data.due_at); // derived: created_at + tenant due_days

// 3. A reviewer at quota rejects your next assign — 409 QUOTA_EXCEEDED.
try {
  await client.request("POST", "/quality/evaluations/assign", {
    form_id: "form_9j2qkx",
    call_id: "call_3m1bcd",
    agent_id: "agent_2k9xd4",
    assignee_id: "user_2f8Kq1XvB",
  });
} catch (err) {
  if (err.code === "QUOTA_EXCEEDED") {
    // Resolve: move the first assignment onto a reviewer with headroom.
    await client.request(
      "POST",
      `/quality/evaluations/${assigned.data.id}/reassign`,
      { assignee_id: "user_7p4wmn" },
    );
  } else {
    throw err;
  }
}

// 4. Read the workload — open counts per reviewer against quota.
const workload = await client.request("GET", "/quality/evaluations/workload");
for (const item of workload.data.items) {
  console.log(`${item.reviewer_id}: ${item.open_count} open, ${item.remaining} left`);
}

from orbit import Orbit

client = Orbit.from_env()  # reads ORBIT_API_KEY

# 1. Turn on sampling: 10% of each agent's calls get queued for review.
client.request("PATCH", "/quality/evaluation-forms/form_9j2qkx",
    json_body={"sample_rate_pct": 10})

# 2. Hand one specific call to a reviewer.
assigned = client.request("POST", "/quality/evaluations/assign", json_body={
    "form_id": "form_9j2qkx",
    "call_id": "call_8h2xq7",
    "agent_id": "agent_2k9xd4",
    "assignee_id": "user_2f8Kq1XvB",
})
print(assigned["data"]["due_at"])  # derived: created_at + tenant due_days

# 3. A reviewer at quota rejects your next assign — 409 QUOTA_EXCEEDED.
#    Resolve by moving the open assignment onto a reviewer with headroom.
try:
    client.request("POST", "/quality/evaluations/assign", json_body={
        "form_id": "form_9j2qkx",
        "call_id": "call_3m1bcd",
        "agent_id": "agent_2k9xd4",
        "assignee_id": "user_2f8Kq1XvB",
    })
except OrbitApiError as exc:
    if exc.code == "QUOTA_EXCEEDED":
        client.request(
            "POST",
            f"/quality/evaluations/{assigned['data']['id']}/reassign",
            json_body={"assignee_id": "user_7p4wmn"},
        )
    else:
        raise

# 4. Read the workload — open counts per reviewer against quota.
workload = client.request("GET", "/quality/evaluations/workload")
for item in workload["data"]["items"]:
    print(f"{item['reviewer_id']}: {item['open_count']} open, {item['remaining']} left")

var assigned map[string]any
err := client.Request(ctx, "PATCH", "/quality/evaluation-forms/form_9j2qkx",
    nil, map[string]any{"sample_rate_pct": 10}, nil, "")
if err != nil {
    return err
}

// 2. Hand one specific call to a reviewer.
assignBody := map[string]any{
    "form_id":     "form_9j2qkx",
    "call_id":     "call_8h2xq7",
    "agent_id":    "agent_2k9xd4",
    "assignee_id": "user_2f8Kq1XvB",
}
err = client.Request(ctx, "POST", "/quality/evaluations/assign",
    nil, assignBody, &assigned, "")
if err != nil {
    // 409 QUOTA_EXCEEDED — resolve by reassigning the open row.
    var orbitErr *orbit.Error
    if errors.As(err, &orbitErr) && orbitErr.Code == "QUOTA_EXCEEDED" {
        id := assigned["data"].(map[string]any)["id"]
        return client.Request(ctx, "POST",
            fmt.Sprintf("/quality/evaluations/%v/reassign", id),
            nil, map[string]any{"assignee_id": "user_7p4wmn"}, nil, "")
    }
    return err
}

// 4. Read the workload — open counts per reviewer against quota.
var workload map[string]any
err = client.Request(ctx, "GET", "/quality/evaluations/workload",
    nil, nil, &workload, "")

The request body is four fields, and everything the server adds is derived — due_at is computed from created_at + due_days in your tenant settings, never stamped:

curl

curl -X POST "https://api.orbit.devotel.io/api/v1/quality/evaluations/assign" \
  -H "X-API-Key: $ORBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"form_id":"form_9j2qkx","call_id":"call_8h2xq7","agent_id":"agent_2k9xd4","assignee_id":"user_2f8Kq1XvB"}'

A successful assign lands as an open (pending, unscored) row:

{
  "data": {
    "id": "qaeval_9x1kzv",
    "form_id": "form_9j2qkx",
    "reviewer_id": "user_2f8Kq1XvB",
    "agent_id": "agent_2k9xd4",
    "call_id": "call_8h2xq7",
    "status": "pending",
    "due_at": "2026-10-16T14:02:31.000Z",
    "created_at": "2026-10-09T14:02:31.000Z",
    "updated_at": "2026-10-09T14:02:31.000Z"
  },
  "meta": { "request_id": "req_4f8tq2w", "timestamp": "2026-10-09T14:02:31.000Z" }
}

The quota rejection is the one error worth handling around this loop — it means the reviewer is full, not that your payload is wrong:

{
  "error": {
    "code": "QUOTA_EXCEEDED",
    "message": "Evaluator is already holding 25 open assignments (quota 25)",
    "status": 409
  },
  "meta": { "request_id": "req_8x2hk9", "timestamp": "2026-10-09T14:02:32.000Z" }
}

A second open assignment on the same call returns 409 ALREADY_ASSIGNED instead — one open row per call, regardless of who holds it. The resolution for quota pressure is POST /api/v1/quality/evaluations/{id}/reassign (shown in the samples above) or a raise of max_open_per_evaluator in the tenant’s assignment settings. The workload read reflects the change on the next call:

{
  "data": {
    "due_days": 7,
    "max_open_per_evaluator": 25,
    "items": [
      { "reviewer_id": "user_2f8Kq1XvB", "open_count": 25, "overdue_count": 3, "quota": 25, "remaining": 0 },
      { "reviewer_id": "user_7p4wmn", "open_count": 4, "overdue_count": 0, "quota": 25, "remaining": 21 }
    ]
  },
  "meta": { "request_id": "req_2j9kvc", "timestamp": "2026-10-09T14:03:10.000Z" }
}

Calibration pass walkthrough

One session, blind submissions, then the agreement report. Create the session pinned to one call and one form — the creator is always on the roster, and if the call already has an AI auto-score, that score lands in the session read-only as ai_seeded: true (the drift baseline the report measures humans against):

Node.js
Python
Go

// 1. Open the session against one call, inviting two reviewers.
const session = await client.request("POST", "/quality/calibrations", {
  form_id: "form_9j2qkx",
  call_id: "call_8h2xq7",
  title: "October compliance calibration",
  reviewer_ids: ["user_2f8Kq1XvB", "user_7p4wmn"],
});
const sessionId = session.data.id; // calib_...

// 2. Each reviewer scores blind — one shot per reviewer, one 201 each.
const submit = await client.request(
  "POST",
  `/quality/calibrations/${sessionId}/scores`,
  { scores: { crit_greeting: 90, crit_identity: 100, crit_pii: 80 } },
);
console.log(submit.data.total_score); // weighted re-derived server-side

// 3. Once every invited reviewer has submitted their own score, read
//    the agreement report (a reviewer who has not submitted gets 403).
const report = await client.request(
  "GET",
  `/quality/calibrations/${sessionId}/report`,
);
console.log(report.data.report.agreement_pct); // 0..100, higher = tighter
console.log(report.data.report.ai_delta?.delta_from_consensus);

// 4. Close the session — it stops accepting new blind scores.
await client.request("POST", `/quality/calibrations/${sessionId}/close`);

# 1. Open the session against one call, inviting two reviewers.
session = client.request("POST", "/quality/calibrations", json_body={
    "form_id": "form_9j2qkx",
    "call_id": "call_8h2xq7",
    "title": "October compliance calibration",
    "reviewer_ids": ["user_2f8Kq1XvB", "user_7p4wmn"],
})
session_id = session["data"]["id"]

# 2. Each reviewer scores blind — one shot per reviewer, one 201 each.
submit = client.request(
    "POST",
    f"/quality/calibrations/{session_id}/scores",
    json_body={"scores": {"crit_greeting": 90, "crit_identity": 100, "crit_pii": 80}},
)
print(submit["data"]["total_score"])  # weighted re-derived server-side

# 3. Once every invited reviewer has submitted their own score, read
#    the agreement report (a reviewer who has not submitted gets 403).
report = client.request("GET", f"/quality/calibrations/{session_id}/report")
print(report["data"]["report"]["agreement_pct"])

# 4. Close the session — it stops accepting new blind scores.
client.request("POST", f"/quality/calibrations/{session_id}/close")

var session map[string]any
sessionBody := map[string]any{
    "form_id":      "form_9j2qkx",
    "call_id":      "call_8h2xq7",
    "title":        "October compliance calibration",
    "reviewer_ids": []string{"user_2f8Kq1XvB", "user_7p4wmn"},
}
err := client.Request(ctx, "POST", "/quality/calibrations",
    nil, sessionBody, &session, "")
if err != nil {
    return err
}
sessionId := session["data"].(map[string]any)["id"]

// 2. Blind submission.
scoreBody := map[string]any{
    "scores": map[string]any{
        "crit_greeting": 90, "crit_identity": 100, "crit_pii": 80,
    },
}
var submit map[string]any
err = client.Request(ctx, "POST",
    fmt.Sprintf("/quality/calibrations/%v/scores", sessionId),
    nil, scoreBody, &submit, "")
if err != nil {
    return err
}

// 3. Read the agreement report, 4. close the session.
var report map[string]any
err = client.Request(ctx, "GET",
    fmt.Sprintf("/quality/calibrations/%v/report", sessionId),
    nil, nil, &report, "")
if err != nil {
    return err
}
err = client.Request(ctx, "POST",
    fmt.Sprintf("/quality/calibrations/%v/close", sessionId),
    nil, nil, nil, "")

curl

curl -X POST "https://api.orbit.devotel.io/api/v1/quality/calibrations" \
  -H "X-API-Key: $ORBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"form_id":"form_9j2qkx","call_id":"call_8h2xq7","title":"October compliance calibration","reviewer_ids":["user_2f8Kq1XvB","user_7p4wmn"]}'

{
  "data": {
    "id": "calib_4t9kqd",
    "form_id": "form_9j2qkx",
    "call_id": "call_8h2xq7",
    "title": "October compliance calibration",
    "status": "open",
    "reviewer_ids": ["user_selfcreator", "user_2f8Kq1XvB", "user_7p4wmn"],
    "ai_seeded": true,
    "created_at": "2026-10-09T14:10:05.000Z",
    "updated_at": "2026-10-09T14:10:05.000Z"
  },
  "meta": { "request_id": "req_7c2vds", "timestamp": "2026-10-09T14:10:05.000Z" }
}

Submitting is one POST /scores per reviewer carrying a raw score map — criterion id to raw 0–100 value. The server re-derives the weighted total against the form definition, so the client never supplies (and cannot override) total_score. Send your score before reading anyone else’s — that is the blind rule:

curl

curl -X POST "https://api.orbit.devotel.io/api/v1/quality/calibrations/calib_4t9kqd/scores" \
  -H "X-API-Key: $ORBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"scores":{"crit_greeting":90,"crit_identity":100,"crit_pii":80}}'

{
  "data": {
    "id": "calibsc_2v8xnq",
    "session_id": "calib_4t9kqd",
    "reviewer_id": "user_2f8Kq1XvB",
    "scores": { "crit_greeting": 90, "crit_identity": 100, "crit_pii": 80 },
    "total_score": 88.4,
    "submitted_at": "2026-10-09T14:12:11.000Z"
  },
  "meta": { "request_id": "req_9d3mmw", "timestamp": "2026-10-09T14:12:11.000Z" }
}

Two submission errors map to the two blind-rule violations: 409 ALREADY_SUBMITTED when you score the same session twice, and 403 FORBIDDEN when you were never invited. Once the roster has submitted, the report opens — with the blind gate still covering stragglers:

{
  "error": {
    "code": "BLIND_SUBMISSION_REQUIRED",
    "message": "Submit your own blind score before viewing the calibration report",
    "status": 403
  },
  "meta": { "request_id": "req_3b8qtv", "timestamp": "2026-10-09T14:12:14.000Z" }
}

The report folds the session detail plus the agreement math — consensus_total is the human mean, mean_pairwise_delta the average pairwise gap, per_criterion the variance per rubric line, reviewer_deltas each grader’s signed distance from consensus, and ai_delta the AI score’s drift from the human consensus (null when no auto-score was seeded):

{
  "data": {
    "session": {
      "id": "calib_4t9kqd",
      "form_id": "form_9j2qkx",
      "call_id": "call_8h2xq7",
      "title": "October compliance calibration",
      "status": "open",
      "created_at": "2026-10-09T14:10:05.000Z",
      "closed_at": null
    },
    "report": {
      "human_submission_count": 3,
      "ready": true,
      "consensus_total": 87.2,
      "mean_pairwise_delta": 4.6,
      "agreement_pct": 95.4,
      "per_criterion": [
        { "criterion_id": "crit_greeting", "label": "Greeting", "mean": 91.3, "variance": 8.2, "stddev": 2.86, "range": 5 },
        { "criterion_id": "crit_identity", "label": "Verified the caller's identity", "mean": 96.7, "variance": 11.1, "stddev": 3.33, "range": 6 },
        { "criterion_id": "crit_pii", "label": "Read the PII disclosure", "mean": 78.0, "variance": 22.7, "stddev": 4.76, "range": 9 }
      ],
      "reviewer_deltas": [
        { "reviewer_id": "user_selfcreator", "total_score": 89.1, "delta_from_consensus": 1.9 },
        { "reviewer_id": "user_2f8Kq1XvB", "total_score": 88.4, "delta_from_consensus": 1.2 },
        { "reviewer_id": "user_7p4wmn", "total_score": 84.1, "delta_from_consensus": -3.1 }
      ],
      "ai_delta": { "reviewer_id": "auto-llm", "total_score": 76.5, "delta_from_consensus": -10.7 }
    }
  },
  "meta": { "request_id": "req_5c8knd", "timestamp": "2026-10-09T14:19:40.000Z" }
}

ready stays false until at least two human submissions are in — before that the numeric fields still render but the report is not yet meaningful. Close the session with POST /calibrations/{id}/close once the pass is complete; a second close returns 409 INVALID_STATE.

Status readback

Two health reads close the loop on automation — coverage answers “how much of my traffic got an AI score”, readiness answers “why not” when that share sags. Both are GETs, both reviewer-scoped, both read-weighted by design.

Node.js
Python
Go

// What share of trailing-24h completed calls did the AI scorecard cover?
const coverage = await client.request("GET", "/quality/autoscore-coverage");
console.log(`${coverage.data.coverage_pct}% of ${coverage.data.completed_calls_last_24h} calls`);

// Why is coverage below target? The generation gates and the 3-day window.
const readiness = await client.request("GET", "/quality/auto-qa-readiness");
console.log(readiness.data.generation_gate);     // toggle + active form
console.log(readiness.data.coverage.coverage_pct); // null when no eligible calls

coverage = client.request("GET", "/quality/autoscore-coverage")
print(coverage["data"]["coverage_pct"])

readiness = client.request("GET", "/quality/auto-qa-readiness")
print(readiness["data"]["generation_gate"])
print(readiness["data"]["coverage"]["coverage_pct"])  # null when no eligible calls

var coverage map[string]any
err := client.Request(ctx, "GET", "/quality/autoscore-coverage", nil, nil, &coverage, "")
if err != nil {
    return err
}

var readiness map[string]any
err = client.Request(ctx, "GET", "/quality/auto-qa-readiness", nil, nil, &readiness, "")
if err != nil {
    return err
}

curl

curl -X GET "https://api.orbit.devotel.io/api/v1/quality/autoscore-coverage" \
  -H "X-API-Key: $ORBIT_API_KEY"

{
  "data": {
    "window": { "hours": 24 },
    "completed_calls_last_24h": 412,
    "auto_scored_calls": 361,
    "coverage_pct": 87.6,
    "avg_auto_score": 83.4,
    "pending_review_rows": 12
  },
  "meta": { "request_id": "req_1h6kga", "timestamp": "2026-10-09T15:02:18.000Z" }
}

pending_review_rows counts the auto-scored rows still flagged for human review — it drains as reviewers override flagged cards. When coverage_pct sags, the readiness snapshot names the gate: a disabled toggle, no active form, or calls without transcripts.

curl

curl -X GET "https://api.orbit.devotel.io/api/v1/quality/auto-qa-readiness" \
  -H "X-API-Key: $ORBIT_API_KEY"

{
  "data": {
    "generation_gate": {
      "enabled": true,
      "has_active_form": true,
      "window_days": 3
    },
    "coverage": {
      "eligible_calls": 1083,
      "auto_scored_calls": 1001,
      "uncovered_calls": 82,
      "coverage_pct": 92.43
    },
    "ai_scored_quality": {
      "avg_total_score": 83.4,
      "flagged_for_review_calls": 41,
      "flagged_share_pct": 4.09
    }
  },
  "meta": { "request_id": "req_6n2qsa", "timestamp": "2026-10-09T15:04:11.000Z" }
}

Both shapes deliberately avoid fake zeros: coverage_pct is null when zero eligible calls exist, and flagged_share_pct is null when zero AI cards exist — an empty state, not a 0% claim. Readiness is the check-in runbook for the automation: if generation_gate.enabled is false the sweep stays idle, if has_active_form is false there is no rubric to score against, and if eligible calls land without transcripts the window narrows before the scorer ever sees them.

Scorecard vocabulary

Three views read the same official-score contract — graded, reviewer-authored evaluations only (an agent scoring their own call, an un-scored queued placeholder, and rows under appeal never enter an official average):

The individual call scorecard — one weighted evaluation of one call against one form. Authored by a human reviewer or by the AI auto-scorer, correctable by override, and acknowledged or appealed by the agent. This is the unit everything else aggregates.
The supervisor scorecard — a per-agent 0–100 composite you define on the fly: custom KPIs (QA average, average handle time, calls handled, CSAT, queue SLA, first-contact resolution) with your own weights and targets, weighted across voice and QA into one number. The definition comes in per request — nothing is stored; the composite is a read-time rollup over the same tables.
QM drivers — the per-criterion rollup: every scorecard line (“Greeting”, “Policy disclosure”, “Resolution”) normalised to a 0–100 average and sorted worst-first, so “which lines move the score” is a read, not a spreadsheet.

Where reads sit, where writes are gated

Every surface above is mounted under /api/v1/quality. The split:

Writes (reviewer scope — owner, admin, or supervisor): assignment and reassignment, calibration sessions and blind scores, composite-scorecard definitions, leaderboard computation, and the scorecard lifecycle itself (author, override, resolve).
Reads: workload, session reports, coaching cards, compliance flags, coverage, readiness, summary, trends, and drivers are all GETs any reviewer can run; agents read only their own card and flags through self-scoped guards.
Blind-gated reads: a calibration report returns 403 to an invited reviewer who has not submitted their own score yet — the blind rule applies to reading agreement, too.

Key routes:

Operation	Endpoint
Assign / reassign a scoring task	`POST /api/v1/quality/evaluations/{assign, :id/reassign}`
Read evaluator workload and quota	`GET /api/v1/quality/evaluations/workload`
Create / list / read a calibration session	`POST	GET /api/v1/quality/calibrations`,` GET /api/v1/quality/calibrations/`
Submit a blind calibration score	`POST /api/v1/quality/calibrations/{id}/scores`
Read the agreement report / close a session	`GET /api/v1/quality/calibrations/{id}/report`, `POST /api/v1/quality/calibrations/{id}/close`
Coaching cards (reviewer rollup or your own)	`GET /api/v1/quality/coaching-cards`
Flagged-for-review compliance queue	`GET /api/v1/quality/compliance-flags`
Auto-QA readiness snapshot	`GET /api/v1/quality/auto-qa-readiness`
Autoscore coverage rollup	`GET /api/v1/quality/autoscore-coverage`
Org-wide QM summary / trends / drivers	`GET /api/v1/quality/{summary, trends, drivers}`
Custom composite supervisor scorecard	`POST /api/v1/quality/scorecards/composite`
Searchable recording + transcript library	`GET /api/v1/quality/recording-library`

The recording library is the worklist all of this draws from: every recording in the tenant, full-text-searchable over finalised diarised transcripts, joined to the linked QA evaluation score and the recording QC verdict — so “recordings where the customer said X, scored below Y, QC-failed” is one query, reviewer-scoped because it surfaces transcript content across all agents. The full request/response shapes are in the Quality API reference; the per-call scorecard pipeline itself is documented in QA evaluations and the performance leaderboard.

Getting Started

Channels

Channels (Voice)

AI Agents

Campaigns

Flows

Numbers & Verify

Concepts

Features

Billing

Webhooks

Security & SSO

Compliance

Legal

Brand & press

Tutorials

Changelog

The call-quality evaluation lifecycle

The call-quality evaluation lifecycle

The cast of entities

Why the quota and blind rules are designed this way

The five-step lifecycle

Sampling via API

Calibration pass walkthrough

Status readback

Scorecard vocabulary

Where reads sit, where writes are gated

​The call-quality evaluation lifecycle

​The cast of entities

​Why the quota and blind rules are designed this way

​The five-step lifecycle

​Sampling via API

​Calibration pass walkthrough

​Status readback

​Scorecard vocabulary

​Where reads sit, where writes are gated

The call-quality evaluation lifecycle

The cast of entities

Why the quota and blind rules are designed this way

The five-step lifecycle

Sampling via API

Calibration pass walkthrough

Status readback

Scorecard vocabulary

Where reads sit, where writes are gated