​

QA sampling: weekly sample assignment

Manual QA scoring has a throughput problem: a human scorer reviewing every conversation a contact center handles is a headcount decision most teams cannot make. The QA sampler removes the manual step — once enabled, a weekly automation draws a random sample of each agent’s recent conversations and assigns them to your evaluators, so the human-review cadence is continuous and the review load stays bounded.

​

Why sampling matters

A quality program that reads every conversation is cost-prohibitive at any meaningful volume, and a program that samples on an ad-hoc basis drifts — reviewers pick conversations, agents game the picks, and the scores stop representing the population. A scheduled random sample fixes both: each agent contributes a fixed quota of conversations drawn at random, and the sample distribution is uniform enough to make per-agent score comparisons fair. Devotel Orbit runs the sampler weekly. Each sweep tops up each active agent to your configured samples-per-agent quota, assigns the sampled conversations round-robin across your evaluators, and writes them into the evaluator inboxes as pending reviews. The sweep is idempotent within the week — re-runs skip agents already at quota, so you never double-sample.

​

Configure it in Settings

Open Settings → QA Sample Assignment in the dashboard (owner or admin role). The group persists to one organization settings block; the shape is: The page is off by default. No samples are created until you explicitly opt in, which keeps a tenant without a QA program from getting review rows it did not ask for.

​

What happens on each sweep

The sampler runs on an hourly cadence but costs one weekly allocation per agent — it tops up agents that fell below quota rather than re-allocating from scratch, so a tenant that opts in mid-week still picks up its quota on the next tick. For each opted-in organization, the sweep:

1. Courts the agent pool: every agent who handled at least one conversation ended in the lookback window is a candidate. Conversations with no human agent on record (pure IVR or bot-only traffic) are never sampled.
2. Checks the existing quota: an agent who already has pending human-review rows at or above your samples-per-agent quota is skipped, and the remaining shortfall tops up.
3. Draws the sample at random: for each under-quota agent the sweep picks the shortfall number of conversations from the lookback window, constrained to conversations with no assigned review yet — AI-auto-scored rows do not block a human sample.
4. Assigns an evaluator: samples are allocated round-robin across the organization members with owner or admin role, sorted so the assignment order is stable across the week. Review load stays evenly spread across the pool.
5. Writes pending review rows: each sampled conversation lands in the assigned evaluator’s inbox as a pending, empty scorecard tied to the organization’s most recently updated active evaluation form.

If no active evaluation form exists, or the organization has no eligible owner/admin members, the sweep skips the organization that tick — no rows are silently parked.

​

Where samples land

Assigned samples appear in Quality → Evaluations, the human-graded scorecard surface. The evaluator assigned to a sample sees it as a pending row against the active evaluation form; the form picker is the most-recently-updated active form, and a second active form is ignored — retire unused forms to pin the canonical one. This surface is distinct from the AI auto-score feed under Quality, which is a different reviewer row tagged as auto-scored. The two can coexist on the same conversation: one carries the AI pre-screen, one carries the human review, and the supervisor reconciles in the evaluation form UI.

​

Tuning the quota

Start at the default of 3 samples per agent per week — three randomly drawn conversations is the smallest sample that gives a usable score distribution without overloading an evaluator. Raise the quota when evaluators finish backlogs early and you want tighter agent-level distributions; lower it when the review queue builds up. The lookback window is the second lever. Keep it at 7 days so reviews stay close to the interaction. Widen it when call volume is thin — a sparse agent pool can hit the weekly quota more often over a 14- or 30-day window than over a strict week.

​

Troubleshooting

Nothing is being sampled. Confirm Enabled is on and an active evaluation form exists. Both conditions are required; the sweep skips an organization quietly when either fails. One evaluator gets all the work. Assignment is round-robin over owner and admin members. If the pool has one eligible member, every sample goes to them; add another owner or admin to spread the load. Sparse agents miss the quota weekly. That is the sampler working correctly against thin volume — no rows are invented to fill it. Widen the lookback window or lower the samples-per-agent quota to match the real call volume.