> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# AI Auto-QA: enable toggle, flag threshold, and where auto-scores land

> Turn on the AI auto-scorer for completed calls, set the flag-threshold that routes low scores to human review, and read the auto-scored evaluations it writes into your QA ledger.

# AI Auto-QA configuration

AI Auto-QA scores every completed agent-handled call against your active evaluation form. A sweep runs once a minute; for each opted-in tenant, it reads the transcript of newly finished calls, sends it to the model with your form's rubric, and writes a structured evaluation row marked `auto_scored: true`. Scores below a threshold you pick are flagged for human review, so your QA team spends its time on the borderline calls instead of manually screening every one.

The feature is off by default. No transcript leaves your tenant database for scoring until you explicitly opt in on **Settings → AI Auto-QA**.

## 1. Enable it

Open **Settings → AI Auto-QA** (owner or admin role). The page exposes one toggle and one slider, persisted under your organization's settings:

| Setting            | What it does                                                                                                                  | Default |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------- | ------- |
| **Enabled**        | Master toggle. Until this is `true`, the sweep skips your tenant entirely and no transcript is sent to the model.             | Off     |
| **Flag threshold** | A score (0–100) below which the auto-scored row is stamped `flagged_for_review: true` and surfaces in the human-review queue. | 70      |

Save the toggle and threshold together. The change takes effect on the sweep's next tick — at most 60 seconds later.

## 2. The rubric it scores against

The sweep does not invent a rubric. It picks your **most recently updated active evaluation form** and flattens its weighted sections and criteria into a prompt. If you have several active forms, retire the ones you don't want driving auto-score; the newest active form is the canonical one.

Author or adjust that form in **Quality → Evaluation forms**, or over `POST /api/v1/quality/evaluation-forms`. The same definition shapes both manual reviewer scoring and auto-scoring, so there is one scorecard to maintain.

If no active form exists, the sweep exits cleanly for your tenant — nothing is scored and nothing is logged as an error.

## 3. What the flag threshold does

A score below your threshold is not deleted or hidden — it's the same row in your QA ledger, with `flagged_for_review: true` set. The "Needs human review" filter in the Quality dashboard reads that flag, so supervisors see a queue of borderline and failing calls to double-check rather than a feed of every call the AI scored.

Lowering the threshold makes the queue smaller and raises the bar for what counts as "worth a human look." Raising it makes the queue larger. The default (70) leans inclusive: a borderline call gets a second look rather than being missed.

You can revise the threshold at any time; only new auto-scored rows pick up the change. Existing rows keep the threshold that was in effect when they were scored.

## 4. How the sweep consumes your setting

The sweep is a once-per-minute tenant fan-out. On each tick it:

1. Loads every org's `qa_autoscore` block as one batch. Only `enabled === true` tenants proceed; everything else is skipped before any per-tenant query runs.
2. Keeps a short-lived per-tick cache (a Redis key shared across the fleet, falling back to a process-local cache) so the settings lookup doesn't hammer the organizations table once per tenant per tick.
3. For each enabled tenant, pre-flights that the QA tables exist, picks the first active evaluation form, then reads completed calls from the last 24 hours that have a transcript and an agent, but no auto-scored evaluation yet.
4. Sends each transcript to the model with the form's rubric and hard-clamps `flag_threshold` to `[0, 100]` (a numeric string, or a missing value, falls back to the default 70) before writing the evaluation row.
5. Writes one `qa_evaluations` row per call with `auto_scored: true` and `flagged_for_review` set from the threshold. The row is idempotent — a partial unique index keyed by the call prevents a second auto row from racing in.

`enabled === true` is the only state the sweep treats as truthy. An absent or `enabled: false` block disables the feature for your tenant with no round-trips.

## 5. Read the auto-scores in the QA ledger

Auto-scored rows land in the same `GET /api/v1/quality/evaluations` ledger as manual ones. Two markers distinguish them:

* `auto_scored: true` — the row came from the sweep, not a human reviewer.
* `reviewer_id` — set to a sentinel value (`auto-llm`) so downstream filters and audit reports can tell auto from human authorship reliably.

Filter your QA ledger by them:

```bash theme={null}
curl "https://api.orbit.devotel.io/api/v1/quality/evaluations?auto_scored=true&flagged_for_review=true" \
  -H "X-API-Key: $ORBIT_API_KEY"
```

The Quality dashboard reads the same endpoint; the "Needs human review" filter maps to `flagged_for_review: true`, and a reviewer's correction on an auto-scored row clears the flag and stamps the reviewer. See the [Quality Management API reference](/api-reference/quality) for the filter matrix and the `POST /api/v1/quality/evaluations/{id}/override` action.

## Troubleshooting

**Nothing is being scored.** Walk the gates in order: the **Enabled** toggle must be on; an active evaluation form must exist (`GET /api/v1/quality/evaluation-forms?is_active=true`); only calls from the last 24 hours with both a transcript and an assigned agent are candidates. Pure-IVR or bot-only calls have no agent to score and are skipped.

**A call was scored but I expected it flagged.** Check the threshold that was in effect when the row was scored. Raising the threshold later does not retroactively flag existing rows.

**The sweep seems slow to start scoring after I toggle it on.** The sweep ticks once a minute and the config-cache TTL is five minutes, so a flip can take up to five minutes to propagate. A call completes, synthesis extracts the transcript, and the autoscore tick picks it up — under two minutes in the normal case, bounded by the cache TTL in the worst case.