Skip to main content

​
AI Auto-QA configuration

AI Auto-QA scores every completed agent-handled call against your active evaluation form. A sweep runs once a minute; for each opted-in tenant, it reads the transcript of newly finished calls, sends it to the model with your form’s rubric, and writes a structured evaluation row marked auto_scored: true. Scores below a threshold you pick are flagged for human review, so your QA team spends its time on the borderline calls instead of manually screening every one. The feature is off by default. No transcript leaves your tenant database for scoring until you explicitly opt in on Settings → AI Auto-QA.

​
1. Enable it

Open Settings → AI Auto-QA (owner or admin role). The page exposes one toggle and one slider, persisted under your organization’s settings: Save the toggle and threshold together. The change takes effect on the sweep’s next tick — at most 60 seconds later.

​
2. The rubric it scores against

The sweep does not invent a rubric. It picks your most recently updated active evaluation form and flattens its weighted sections and criteria into a prompt. If you have several active forms, retire the ones you don’t want driving auto-score; the newest active form is the canonical one. Author or adjust that form in Quality → Evaluation forms, or over POST /api/v1/quality/evaluation-forms. The same definition shapes both manual reviewer scoring and auto-scoring, so there is one scorecard to maintain. If no active form exists, the sweep exits cleanly for your tenant — nothing is scored and nothing is logged as an error.

​
3. What the flag threshold does

A score below your threshold is not deleted or hidden — it’s the same row in your QA ledger, with flagged_for_review: true set. The “Needs human review” filter in the Quality dashboard reads that flag, so supervisors see a queue of borderline and failing calls to double-check rather than a feed of every call the AI scored. Lowering the threshold makes the queue smaller and raises the bar for what counts as “worth a human look.” Raising it makes the queue larger. The default (70) leans inclusive: a borderline call gets a second look rather than being missed. You can revise the threshold at any time; only new auto-scored rows pick up the change. Existing rows keep the threshold that was in effect when they were scored.

​
4. How the sweep consumes your setting

The sweep is a once-per-minute tenant fan-out. On each tick it:
  1. Loads every org’s qa_autoscore block as one batch. Only enabled === true tenants proceed; everything else is skipped before any per-tenant query runs.
  2. Keeps a short-lived per-tick cache (a Redis key shared across the fleet, falling back to a process-local cache) so the settings lookup doesn’t hammer the organizations table once per tenant per tick.
  3. For each enabled tenant, pre-flights that the QA tables exist, picks the first active evaluation form, then reads completed calls from the last 24 hours that have a transcript and an agent, but no auto-scored evaluation yet.
  4. Sends each transcript to the model with the form’s rubric and hard-clamps flag_threshold to [0, 100] (a numeric string, or a missing value, falls back to the default 70) before writing the evaluation row.
  5. Writes one qa_evaluations row per call with auto_scored: true and flagged_for_review set from the threshold. The row is idempotent — a partial unique index keyed by the call prevents a second auto row from racing in.
enabled === true is the only state the sweep treats as truthy. An absent or enabled: false block disables the feature for your tenant with no round-trips.

​
5. Read the auto-scores in the QA ledger

Auto-scored rows land in the same GET /api/v1/quality/evaluations ledger as manual ones. Two markers distinguish them:
  • auto_scored: true — the row came from the sweep, not a human reviewer.
  • reviewer_id — set to a sentinel value (auto-llm) so downstream filters and audit reports can tell auto from human authorship reliably.
Filter your QA ledger by them:
The Quality dashboard reads the same endpoint; the “Needs human review” filter maps to flagged_for_review: true, and a reviewer’s correction on an auto-scored row clears the flag and stamps the reviewer. See the Quality Management API reference for the filter matrix and the POST /api/v1/quality/evaluations/{id}/override action.

​
Troubleshooting

Nothing is being scored. Walk the gates in order: the Enabled toggle must be on; an active evaluation form must exist (GET /api/v1/quality/evaluation-forms?is_active=true); only calls from the last 24 hours with both a transcript and an assigned agent are candidates. Pure-IVR or bot-only calls have no agent to score and are skipped. A call was scored but I expected it flagged. Check the threshold that was in effect when the row was scored. Raising the threshold later does not retroactively flag existing rows. The sweep seems slow to start scoring after I toggle it on. The sweep ticks once a minute and the config-cache TTL is five minutes, so a flip can take up to five minutes to propagate. A call completes, synthesis extracts the transcript, and the autoscore tick picks it up — under two minutes in the normal case, bounded by the cache TTL in the worst case.