{ "data": { "window": { "hours": 24 }, "completed_calls_last_24h": 412, "auto_scored_calls": 387, "coverage_pct": 93.9, "avg_auto_score": 81.4, "pending_review_rows": 12 }, "meta": { "request_id": "req_01HZQX4E7JQ4M2E2H2DM9XE5FJ", "timestamp": "2026-09-12T12:00:00.000Z" } }

AI Auto-QA configuration

AI Auto-QA scores every completed agent-handled call against your active evaluation form. A sweep runs once a minute; for each opted-in tenant, it reads the transcript of newly finished calls, sends it to the model with your form’s rubric, and writes a structured evaluation row marked auto_scored: true. Scores below a threshold you pick are flagged for human review, so your QA team spends its time on the borderline calls instead of manually screening every one. The feature is off by default. No transcript leaves your tenant database for scoring until you explicitly opt in on Settings → AI Auto-QA.

1. Enable it

Open Settings → AI Auto-QA (owner or admin role). The page exposes one toggle and one slider, persisted under your organization’s settings:

Setting	What it does	Default
Enabled	Master toggle. Until this is `true`, the sweep skips your tenant entirely and no transcript is sent to the model.	Off
Flag threshold	A score (0–100) below which the auto-scored row is stamped `flagged_for_review: true` and surfaces in the human-review queue.	70

Save the toggle and threshold together. The change takes effect on the sweep’s next tick — at most 60 seconds later.

2. The rubric it scores against

The sweep does not invent a rubric. It picks your most recently updated active evaluation form and flattens its weighted sections and criteria into a prompt. If you have several active forms, retire the ones you don’t want driving auto-score; the newest active form is the canonical one. Author or adjust that form in Quality → Evaluation forms, or over POST /api/v1/quality/evaluation-forms. The same definition shapes both manual reviewer scoring and auto-scoring, so there is one scorecard to maintain. If no active form exists, the sweep exits cleanly for your tenant — nothing is scored and nothing is logged as an error.

3. What the flag threshold does

A score below your threshold is not deleted or hidden — it’s the same row in your QA ledger, with flagged_for_review: true set. The “Needs human review” filter in the Quality dashboard reads that flag, so supervisors see a queue of borderline and failing calls to double-check rather than a feed of every call the AI scored. Lowering the threshold makes the queue smaller and raises the bar for what counts as “worth a human look.” Raising it makes the queue larger. The default (70) leans inclusive: a borderline call gets a second look rather than being missed. You can revise the threshold at any time; only new auto-scored rows pick up the change. Existing rows keep the threshold that was in effect when they were scored.

4. How the sweep consumes your setting

The sweep is a once-per-minute tenant fan-out. On each tick it:

Loads every org’s qa_autoscore block as one batch. Only enabled === true tenants proceed; everything else is skipped before any per-tenant query runs.
Keeps a short-lived per-tick cache (a Redis key shared across the fleet, falling back to a process-local cache) so the settings lookup doesn’t hammer the organizations table once per tenant per tick.
For each enabled tenant, pre-flights that the QA tables exist, picks the first active evaluation form, then reads completed calls from the last 24 hours that have a transcript and an agent, but no auto-scored evaluation yet.
Sends each transcript to the model with the form’s rubric and hard-clamps flag_threshold to [0, 100] (a numeric string, or a missing value, falls back to the default 70) before writing the evaluation row.
Writes one qa_evaluations row per call with auto_scored: true and flagged_for_review set from the threshold. The row is idempotent — a partial unique index keyed by the call prevents a second auto row from racing in.

enabled === true is the only state the sweep treats as truthy. An absent or enabled: false block disables the feature for your tenant with no round-trips.

5. Read the auto-scores in the QA ledger

Auto-scored rows land in the same GET /api/v1/quality/evaluations ledger as manual ones. Two markers distinguish them:

auto_scored: true — the row came from the sweep, not a human reviewer.
reviewer_id — set to a sentinel value (auto-llm) so downstream filters and audit reports can tell auto from human authorship reliably.

Filter your QA ledger by them:

cURL
Node.js SDK
Python SDK
Go SDK

curl "https://api.orbit.devotel.io/api/v1/quality/evaluations?auto_scored=true&flagged_for_review=true" \
  -H "X-API-Key: $ORBIT_API_KEY"

import { Orbit } from '@devotel-orbit/node';

interface EvaluationRow {
  id: string;
  call_id: string | null;
  agent_id: string | null;
  form_id: string;
  reviewer_id: string;
  scores: Record<string, number>;
  total_score: number | null;
  auto_scored: boolean;
  flagged_for_review: boolean;
  created_at: string;
}

const orbit = new Orbit({ apiKey: process.env.ORBIT_API_KEY! });

const { data } = await orbit.request<{
  items: EvaluationRow[];
  next_cursor: string | null;
}>(
  'GET',
  '/api/v1/quality/evaluations?auto_scored=true&flagged_for_review=true',
);

from orbit_sdk import OrbitClient

client = OrbitClient.from_api_key()

response = client.request(
    "GET",
    "/api/v1/quality/evaluations",
    params={"auto_scored": "true", "flagged_for_review": "true"},
)
rows = response["data"]["items"]

package main

import (
	"context"
	"fmt"
	"net/url"

	orbit "github.com/devotel/orbit-sdk-go/orbit"
)

func main() {
	client := orbit.NewClient()
	query := url.Values{
		"auto_scored":        []string{"true"},
		"flagged_for_review": []string{"true"},
	}
	var out struct {
		Data struct {
			Items []map[string]any `json:"items"`
		} `json:"data"`
	}
	if err := client.Request(context.Background(), "GET",
		"/api/v1/quality/evaluations", query, nil, &out, ""); err != nil {
		panic(err)
	}
	fmt.Println(out.Data.Items)
}

The SDK helpers above are the generic escape hatch (orbit.request / client.request / client.Request) — they work for every Quality endpoint, typed or not, so you are never blocked waiting on a helper method.

Drain the whole queue, one cursor at a time

The ledger is keyset-paginated: each response carries a data.next_cursor token, and you follow it with the cursor query parameter until the token comes back null. The loop below pages through the flagged auto-scored rows so an export or a review-pipeline consumer gets every row, not just page one:

let cursor: string | null = null;
const pending: EvaluationRow[] = [];

do {
  const qs = new URLSearchParams({
    auto_scored: 'true',
    flagged_for_review: 'true',
  });
  if (cursor) qs.set('cursor', cursor);

  const { data } = await orbit.request<{
    items: EvaluationRow[];
    next_cursor: string | null;
  }>('GET', `/api/v1/quality/evaluations?${qs.toString()}`);

  pending.push(...data.items);
  cursor = data.next_cursor;
} while (cursor);

Hand a malformed cursor back and the endpoint returns 400 INVALID_CURSOR, so page tokens should be treated as opaque.

Pre-flight readiness before you dash

GET /api/v1/quality/auto-qa-readiness is the sweep’s pre-flight check given to you as an endpoint. It answers all three gates at once — toggle on, active form present, and windowed coverage — so you validate setup from any language before you watch the dashboard.

cURL
Node.js SDK
Python SDK
Go SDK

curl "https://api.orbit.devotel.io/api/v1/quality/auto-qa-readiness" \
  -H "X-API-Key: $ORBIT_API_KEY"

const { data } = await orbit.request<{
  generation_gate: {
    enabled: boolean;
    has_active_form: boolean;
    window_days: number;
  };
  coverage: {
    eligible_calls: number;
    auto_scored_calls: number;
    uncovered_calls: number;
    coverage_pct: number | null;
  };
  ai_scored_quality: {
    avg_total_score: number | null;
    flagged_for_review_calls: number;
    flagged_share_pct: number | null;
  };
}>('GET', '/api/v1/quality/auto-qa-readiness');

if (!data.generation_gate.enabled || !data.generation_gate.has_active_form) {
  // Turn the toggle on, or retire to an active evaluation form first.
}

response = client.request("GET", "/api/v1/quality/auto-qa-readiness")
gate = response["data"]["generation_gate"]

var out struct {
	Data struct {
		GenerationGate struct {
			Enabled       bool `json:"enabled"`
			HasActiveForm bool `json:"has_active_form"`
		} `json:"generation_gate"`
	} `json:"data"`
}
err := client.Request(context.Background(), "GET",
	"/api/v1/quality/auto-qa-readiness", nil, nil, &out, "")

Read it as three blocks: generation_gate tells you whether the sweep would even pick you (toggle + form); coverage sizes how much of the eligible window traffic already has an AI scorecard; ai_scored_quality is what the AI produced over its own rows. A gate that reads enabled: false or has_active_form: false explains a coverage_pct: null — fix the gate before reading coverage.

The readiness card

The same snapshot the pre-flight endpoint returns now renders as the “AI auto-QA scorecard coverage” card on the Quality hub — the generation gates as a badge, the coverage gauge, the average AI score, and the flagged share, with a next-step chip (“Turn on AI auto-QA” / “Activate a scorecard form”) for whichever gate is missing. Check the card before you investigate “nothing is being scored”; a missing gate explains a coverage_pct: null on the UI as well as on the API. See AI auto-QA readiness for the full card walk-through.

Check coverage before debugging “nothing is being scored”

The sweep’s once-a-minute tick and the “nothing is scored” failure mode are easiest to confirm with the coverage rollup instead of eyeballing the ledger. GET /api/v1/quality/autoscore-coverage returns, for the trailing 24 hours, how many calls completed, how many of them carry an auto-scored row, the share covered, the mean auto score, and how many flagged rows are still waiting on a human:

cURL
Node.js SDK
Python SDK
Go SDK

curl "https://api.orbit.devotel.io/api/v1/quality/autoscore-coverage" \
  -H "X-API-Key: $ORBIT_API_KEY"

const { data } = await orbit.request<{
  window: { hours: number };
  completed_calls_last_24h: number;
  auto_scored_calls: number;
  coverage_pct: number | null;
  avg_auto_score: number | null;
  pending_review_rows: number;
}>('GET', '/api/v1/quality/autoscore-coverage');

response = client.request("GET", "/api/v1/quality/autoscore-coverage")
coverage = response["data"]

var out struct {
	Data struct {
		CoveragePct  *float64 `json:"coverage_pct"`
		AvgAutoScore *float64 `json:"avg_auto_score"`
	} `json:"data"`
}
err := client.Request(context.Background(), "GET",
	"/api/v1/quality/autoscore-coverage", nil, nil, &out, "")

{
  "data": {
    "window": { "hours": 24 },
    "completed_calls_last_24h": 412,
    "auto_scored_calls": 387,
    "coverage_pct": 93.9,
    "avg_auto_score": 81.4,
    "pending_review_rows": 12
  },
  "meta": {
    "request_id": "req_01HZQX4E7JQ4M2E2H2DM9XE5FJ",
    "timestamp": "2026-09-12T12:00:00.000Z"
  }
}

Read the response in order:

completed_calls_last_24h is the denominator. If it is 0, no calls finished in the window — check call volume, not the scorer.
coverage_pct compares completed calls against calls that actually received an auto row. A low value means the scorer’s gates are excluding traffic: no active evaluation form, calls without transcripts, or calls without an assigned agent (the gates in step 2 and Troubleshooting).
avg_auto_score is null when nothing was scored, so do not read “no score” as “scored zero.”
pending_review_rows counts flagged auto rows nobody has overridden yet — the human-review queue this guide wires into the dashboard.

The Quality dashboard reads the same endpoint; the “Needs human review” filter maps to flagged_for_review: true, and a reviewer’s correction on an auto-scored row clears the flag and stamps the reviewer. See the Quality Management API reference for the filter matrix and the POST /api/v1/quality/evaluations/{id}/override action.

Troubleshooting

Nothing is being scored. Start with GET /api/v1/quality/autoscore-coverage (steps below) — it tells you whether the denominator (completed_calls_last_24h) is zero or the gates are excluding traffic. Then walk the gates in order: the Enabled toggle must be on; an active evaluation form must exist (GET /api/v1/quality/evaluation-forms?is_active=true); only calls from the last 24 hours with both a transcript and an assigned agent are candidates. Pure-IVR or bot-only calls have no agent to score and are skipped. A call was scored but I expected it flagged. Check the threshold that was in effect when the row was scored. Raising the threshold later does not retroactively flag existing rows. The sweep seems slow to start scoring after I toggle it on. The sweep ticks once a minute and the config-cache TTL is five minutes, so a flip can take up to five minutes to propagate. A call completes, synthesis extracts the transcript, and the autoscore tick picks it up — under two minutes in the normal case, bounded by the cache TTL in the worst case.

WhatsApp

Developer console

Contact center

Guides

Messaging Controls

Audience & Support Tools

AI Auto-QA: enable toggle, flag threshold, and where auto-scores land

AI Auto-QA configuration

1. Enable it

2. The rubric it scores against

3. What the flag threshold does

4. How the sweep consumes your setting

5. Read the auto-scores in the QA ledger

Drain the whole queue, one cursor at a time

Pre-flight readiness before you dash

The readiness card

Check coverage before debugging “nothing is being scored”

Troubleshooting

​AI Auto-QA configuration

​1. Enable it

​2. The rubric it scores against

​3. What the flag threshold does

​4. How the sweep consumes your setting

​5. Read the auto-scores in the QA ledger

​Drain the whole queue, one cursor at a time

​Pre-flight readiness before you dash

​The readiness card

​Check coverage before debugging “nothing is being scored”

​Troubleshooting

AI Auto-QA configuration

1. Enable it

2. The rubric it scores against

3. What the flag threshold does

4. How the sweep consumes your setting

5. Read the auto-scores in the QA ledger

Drain the whole queue, one cursor at a time

Pre-flight readiness before you dash

The readiness card

Check coverage before debugging “nothing is being scored”

Troubleshooting