> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Continuous production-call eval sampling for AI voice agents

> Sample a deterministic percent of finished production calls into voice-eval runs, so your golden sets keep scoring live traffic long after launch — without re-running every call.

# Continuous production-call eval sampling

Offline evaluation tells you the agent passed your test suite. Production sampling tells you the agent still passes on real callers. Turn it on and a deterministic slice of every completed production call is replayed against a golden set as a background eval run — a standing quality check on live traffic.

The endpoint paths below are relative. Send them against
`https://api.orbit.devotel.io/api/v1`.

## What production sampling is

When an AI voice call completes, a sampling gate asks two questions:

1. Does this call's agent match the policy — and is the policy enabled?
2. Which bucket does this call hash into?

The gate hashes a stable call key into one of 100 buckets. If the bucket falls below `sample_percent`, the call is sampled and a voice-eval run is started against the configured golden set. Two properties matter for operators:

* **Deterministic.** The same call always lands in the same bucket, across retries and process restarts. A sampled call is never double-run, and a rejected call never sneaks in later. Raising `sample_percent` from 10 to 25 strictly expands the set of sampled calls — it never shuffles them.
* **Config, not data.** The policy is a JSON blob stored in your organization's AI voice agent settings. There is no table to provision and nothing to migrate; a tenant that never sets a policy simply keeps the safe default (disabled, 0%).

Only calls that complete normally enter the funnel. Dropped or failed setup attempts do not produce eval runs.

## Policy fields

The policy is a single document with four fields:

| Field            | Type                   | Required when enabled | Notes                                                                                                                                                                                            |
| ---------------- | ---------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `enabled`        | boolean                | —                     | Master switch. `false` short-circuits every decision; `sample_percent` is then irrelevant.                                                                                                       |
| `sample_percent` | number `0–100`         | —                     | Percent of completed calls routed into eval runs. `0` samples nothing even when enabled. Use whole percents; `10` means roughly one call in ten.                                                 |
| `agent_ids`      | string array or `null` | no                    | `null` (or omitted) applies the policy to **every** AI voice agent in the workspace. A list restricts sampling to those agent ids — duplicate ids are rejected and the list caps at 500 entries. |
| `golden_set_id`  | string or `null`       | **yes**               | The golden set sampled runs execute. Enabling a policy without one is rejected with a `400` — an enabled policy with no golden set would silently do nothing.                                    |

An example that samples 10% of calls on two specific agents:

```json theme={null}
{
  "enabled": true,
  "sample_percent": 10,
  "agent_ids": ["agt_support_frontline", "agt_billing_assist"],
  "golden_set_id": "vevg_support_hallmark_v3"
}
```

A workspace-wide stance that samples 5% everywhere:

```json theme={null}
{
  "enabled": true,
  "sample_percent": 5,
  "agent_ids": null,
  "golden_set_id": "vevg_all_agents_baseline"
}
```

## Route surface

Two endpoints manage the policy, both under the voice-eval prefix and both gated to the `owner`, `admin`, and `developer` workspace roles:

* `GET /agents/voice-eval/sampling-policy` — read the current policy.
* `PUT /agents/voice-eval/sampling-policy` — replace it.

PUT is a **whole-document upsert**: the body you send becomes the entire policy. Fields you omit fall back to their defaults (`agent_ids` → `null` = all agents, `golden_set_id` → `null`), so send all four fields every time — a partial body does not preserve omitted state beyond those defaults.

The response is always the normalized, fully-populated policy — the same document the sampling gate evaluates — shaped so you can round-trip it back in the next PUT:

```json theme={null}
{
  "data": {
    "enabled": true,
    "sample_percent": 10,
    "agent_ids": ["agt_support_frontline", "agt_billing_assist"],
    "golden_set_id": "vevg_support_hallmark_v3"
  },
  "meta": { "request_id": "req_…", "timestamp": "2026-08-25T12:00:00.000Z" }
}
```

Validation failures return `400` with `VALIDATION_ERROR` — including the two guard rails: duplicate agent ids, and `enabled: true` with no `golden_set_id`.

## Set it up

1. **Create or identify the golden set.** Production sampling reuses the same golden sets your pre-launch evaluation uses — see the [voice-evaluation endpoints](/api-reference/endpoints/agents#voice-evaluation) for creating one (`POST /agents/voice-eval/golden-sets`).

2. **Read the current policy** to see what is in effect:

   ```bash theme={null}
   curl -X GET https://api.orbit.devotel.io/api/v1/agents/voice-eval/sampling-policy \
     -H "X-API-Key: dv_live_sk_..."
   ```

3. **Write the policy.** Start small — 5% workspace-wide, or a higher percent restricted to one agent — and confirm the runs appear:

   ```bash theme={null}
   curl -X PUT https://api.orbit.devotel.io/api/v1/agents/voice-eval/sampling-policy \
     -H "X-API-Key: dv_live_sk_..." \
     -H "Content-Type: application/json" \
     -d '{
       "enabled": true,
       "sample_percent": 5,
       "agent_ids": null,
       "golden_set_id": "vevg_all_agents_baseline"
     }'
   ```

4. **Watch the eval runs.** Sampled calls surface as ordinary eval runs (`GET /agents/voice-eval/runs`), scored by the same judge as your offline runs. Filter by `golden_set_id` to see what production sampling produced.

To stop sampling, write back the empty stance:

```bash theme={null}
curl -X PUT https://api.orbit.devotel.io/api/v1/agents/voice-eval/sampling-policy \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": false,
    "sample_percent": 0,
    "agent_ids": null,
    "golden_set_id": null
  }'
```

## Ops notes

* **Unconfigured workspaces read a safe default.** If the workspace has never written a policy, `GET` returns the empty stance (`enabled: false`, `sample_percent: 0`, `agent_ids: null`, `golden_set_id: null`) with a `200` — never a `404`. You can treat a fetched policy as always present.
* **Sampled calls are eval runs, not a side channel.** A production sample lands in the same run list, judge, and regression-delta machinery as a manually triggered run. Anything you build on eval runs — dashboards, alerts, rollout gates — sees production samples automatically.
* **Sampling works alongside rollout, not instead of it.** Use offline gates ([agent versions](/agents/agent-versions), regression tests, persona simulation) to qualify a candidate, the canary ladder in the [rollout pipeline](/guides/ai-agent-rollout-pipeline) to promote it, and production sampling to keep scoring the steady state afterward. Pinning a policy to one agent id is the natural way to hold 100% eval coverage on a canary candidate while the rest of the fleet samples at a low percent.
* **Percent changes take effect on the next completed call.** The gate reads the current policy per call; there is no cache to flush.

## Related pages

* [Agent versions](/agents/agent-versions) — the snapshots eval results reference.
* [Safely Roll Out an AI Agent](/guides/ai-agent-rollout-pipeline) — the canary stages and promotion gates that consume the same eval runs.
* [Voice evaluation endpoints](/api-reference/endpoints/agents#voice-evaluation) — golden sets and eval runs in full.