Skip to main content

​
Continuous production-call eval sampling

Offline evaluation tells you the agent passed your test suite. Production sampling tells you the agent still passes on real callers. Turn it on and a deterministic slice of every completed production call is replayed against a golden set as a background eval run — a standing quality check on live traffic. The endpoint paths below are relative. Send them against https://api.orbit.devotel.io/api/v1.

​
What production sampling is

When an AI voice call completes, a sampling gate asks two questions:
  1. Does this call’s agent match the policy — and is the policy enabled?
  2. Which bucket does this call hash into?
The gate hashes a stable call key into one of 100 buckets. If the bucket falls below sample_percent, the call is sampled and a voice-eval run is started against the configured golden set. Two properties matter for operators:
  • Deterministic. The same call always lands in the same bucket, across retries and process restarts. A sampled call is never double-run, and a rejected call never sneaks in later. Raising sample_percent from 10 to 25 strictly expands the set of sampled calls — it never shuffles them.
  • Config, not data. The policy is a JSON blob stored in your organization’s AI voice agent settings. There is no table to provision and nothing to migrate; a tenant that never sets a policy simply keeps the safe default (disabled, 0%).
Only calls that complete normally enter the funnel. Dropped or failed setup attempts do not produce eval runs.

​
Policy fields

The policy is a single document with four fields: An example that samples 10% of calls on two specific agents:
A workspace-wide stance that samples 5% everywhere:

​
Route surface

Two endpoints manage the policy, both under the voice-eval prefix and both gated to the owner, admin, and developer workspace roles:
  • GET /agents/voice-eval/sampling-policy — read the current policy.
  • PUT /agents/voice-eval/sampling-policy — replace it.
PUT is a whole-document upsert: the body you send becomes the entire policy. Fields you omit fall back to their defaults (agent_ids → null = all agents, golden_set_id → null), so send all four fields every time — a partial body does not preserve omitted state beyond those defaults. The response is always the normalized, fully-populated policy — the same document the sampling gate evaluates — shaped so you can round-trip it back in the next PUT:
Validation failures return 400 with VALIDATION_ERROR — including the two guard rails: duplicate agent ids, and enabled: true with no golden_set_id.

​
Set it up

  1. Create or identify the golden set. Production sampling reuses the same golden sets your pre-launch evaluation uses — see the voice-evaluation endpoints for creating one (POST /agents/voice-eval/golden-sets).
  2. Read the current policy to see what is in effect:
  3. Write the policy. Start small — 5% workspace-wide, or a higher percent restricted to one agent — and confirm the runs appear:
  4. Watch the eval runs. Sampled calls surface as ordinary eval runs (GET /agents/voice-eval/runs), scored by the same judge as your offline runs. Filter by golden_set_id to see what production sampling produced.
To stop sampling, write back the empty stance:

​
Ops notes

  • Unconfigured workspaces read a safe default. If the workspace has never written a policy, GET returns the empty stance (enabled: false, sample_percent: 0, agent_ids: null, golden_set_id: null) with a 200 — never a 404. You can treat a fetched policy as always present.
  • Sampled calls are eval runs, not a side channel. A production sample lands in the same run list, judge, and regression-delta machinery as a manually triggered run. Anything you build on eval runs — dashboards, alerts, rollout gates — sees production samples automatically.
  • Sampling works alongside rollout, not instead of it. Use offline gates (agent versions, regression tests, persona simulation) to qualify a candidate, the canary ladder in the rollout pipeline to promote it, and production sampling to keep scoring the steady state afterward. Pinning a policy to one agent id is the natural way to hold 100% eval coverage on a canary candidate while the rest of the fleet samples at a low percent.
  • Percent changes take effect on the next completed call. The gate reads the current policy per call; there is no cache to flush.