> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent evals: datasets, runs, and pass-rate gates

> Build an eval dataset from real conversations or scripted rows, run it against an agent version with a judge rubric, read the failures, and gate a rollout on the pass rate.

# Agent evals: datasets, runs, and pass-rate gates

An eval run replays a fixed dataset of inputs against an agent version and
lets an LLM judge score every response. Because the dataset and the rubric
stay pinned, two runs compare cleanly — which is what makes a pass rate safe
to gate a rollout on. This guide walks the full loop: declare a dataset, run
it, triage the failures, and use the result as a gate.

Endpoint paths below are relative. Send them against
`https://api.orbit.devotel.io/api/v1`. All routes need
`agents:read` / `agents:write` scope.

## 1. What an eval run expects

Every run binds three things, and the run row records all three so results
stay attributable:

* **A dataset** — up to 1,000 rows of `{ input, expected_output, metadata }`.
  The runner sends each `input` to the agent and the judge compares the
  agent's `actual_output` against your `expected_output`.
* **A rubric** — one of `correctness`, `helpfulness`, `safety`,
  `groundedness` (faithfulness against retrieved knowledge-base context), or
  `custom` (your own `rubric_prompt`, required and up to 8,000 characters
  when you pick `custom`).
* **A judge model** — the clause model that grades each row. Omit
  `judge_model` to inherit the default (`claude-haiku-4-5-20251001`); when
  you pin one it must be an approved Claude model id or the run is rejected
  at submit with a `422`.

Runs always execute against a saved agent version. If you are testing a
candidate, save it first — see [Agent versions](/agents/agent-versions). For
how production sampling keeps eval coverage on live traffic after launch,
see [Continuous production evals](/agents/continuous-production-evals).

## 2. Declare a dataset

Create datasets from the **Agents → your agent → Evals → Datasets** page, or
with `POST /agents/{agentId}/evals/datasets`. Two event sources feed a
dataset; use both.

### Scripted rows

Write rows by hand when you know the case you want covered — a canonical
support question, a regression you just fixed, an edge case from a ticket.

```bash theme={null}
curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/datasets \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "refund-golden-v1",
    "description": "Refund policy cases — quarter 3",
    "rows": [
      {
        "input": "You charged me twice for order 42. Refund the duplicate.",
        "expected_output": "Confirms the duplicate charge and initiates a refund with a clear timeline.",
        "metadata": { "source": "scripted", "case": "duplicate-charge" }
      }
    ]
  }'
```

The response is `201` with the dataset `id` and `row_count`. At least one
row is required; the cap is 1,000 per dataset.

### Promoted production conversations

The richer source is live traffic. One turn of a captured conversation's
execution trace becomes one `{ input: user prompt, expected_output:
assistant response }` row:

```bash theme={null}
curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/datasets/from-conversation \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "conversation_id": "conv_9f2a...",
    "name": "refund-failures-captured",
    "limit": 50
  }'
```

`include_non_success_turns` defaults to `true` — on purpose, because
capturing the turns where the agent's audit outcome was a failure is exactly
what you want in the golden set. Narrow the window with `limit` (defaults to
the most recent turns, max 200).

Read back what you have with `GET /agents/{agentId}/evals/datasets` (list)
and `GET /agents/{agentId}/evals/datasets/{dsId}` (rows). Dataset names and
rows are editable later — rename a set, fix a mislabeled
`expected_output`, or prune a stale case.

## 3. Run a suite

Start a run from the **Evals → Runs** page or with
`POST /agents/{agentId}/evals/runs`:

```bash theme={null}
curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/runs \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_id": "evalDs_abc",
    "rubric_name": "correctness",
    "threshold": 80
  }'
```

The response is `202` with `status: "pending"` — the run executes in the
background. Poll `GET /agents/{agentId}/evals/runs/{runId}` until the
`status` flips to `completed` (or `failed`), then read the aggregate
`passed_rows`, `failed_rows`, and `avg_score` on the same object.
`GET /agents/{agentId}/evals/runs` lists recent runs; filter with
`?status=completed`.

Pick a rubric that matches the failure you are guarding against. For a
knowledge-grounded agent, `groundedness` is the stronger check — it catches
answers that read well but are not in the retrieval context.

## 4. Read the failures

The run detail returns a `results` array (up to 1,000 rows) with, per row:
`actual_output`, the judge's `score`, a boolean `passed`, and
`judge_reasoning`, plus `latency_ms` and `cost_cents`. Triage in this order:

1. **Filter to `passed: false`.** Each failed row carries the exact
   `dataset_row_id`, so you always know which input regressed.
2. **Read `judge_reasoning` before touching the prompt.** The judge names
   which rubric item failed — "no refund timeline given" is actionable;
   "score 62" alone is not.
3. **Check `error_message`.** A row that errored (runtime fault, not a judge
   reject) shows the cause in the row itself; fix the fault before treating
   it as a quality regression.
4. **Compare against the baseline run.** `GET
   /agents/{agentId}/evals/runs/compare?run_a_id=<old>&run_b_id=<new>`
   reports the same verdict on the same rows — it rejects with a
   `400 EVAL_COMPARE_DATASET_MISMATCH` if the two runs were not run against
   the same dataset.

For knowledge-grounded failures, the route to fix is the knowledge source,
not the prompt — the retrieval benchmark in the
[rollout pipeline guide](/guides/ai-agent-rollout-pipeline) isolates that.

## 5. Gate on a pass-rate threshold

Each run carries a `threshold` (0–100, default 70): the judge score a row
must reach to count as passed. Set it at submit time and keep it pinned
across runs so pass rates move only because the agent moved.

Wire the gate like this:

1. Run the pinned dataset against the candidate version.
2. Read `passed_rows` / `total_rows`; require the pass rate to clear your
   gate (for example, all rows passed, or at least 90% on a noisy rubric).
3. Promote only then — via the canary ladder in the
   [rollout pipeline guide](/guides/ai-agent-rollout-pipeline), never by
   hand.

A CI job does the same thing mechanically: poll
`GET /agents/{agentId}/evals/runs/{runId}` until `completed`, then assert
`passed_rows / total_rows` clears the gate before promoting. The threshold
stays a row-level judge cutoff; the gate asserts on the aggregate.

## 6. Holdout vs experiment

Keep a holdout when you experiment.

* **Freeze a baseline.** Save the current production prompt as a version and
  never edit it. Run the pinned dataset against that version once and keep
  the run id as your reference.
* **Score experiments against the holdout.** An experiment splits live
  traffic between the control version (variant A) and the candidate
  (variant B). Run the same dataset against both versions and use
  `GET /agents/{agentId}/evals/runs/compare` for the verdict — never compare
  a candidate against a dataset that drifted since the baseline was scored.
* **Delete stales.** A baseline run on a dataset you have since edited is a
  false holdout. Re-run the baseline after any dataset change, and record
  both run ids on the experiment.

## 7. Failure modes

* **A run stuck at `pending`.** The background runner never picked it up —
  re-submit and check `GET /agents/{agentId}/evals/runs?status=failed` for
  an `error_message` on the sibling rows. A run that never terminates almost
  always carries its cause on the failed run detail.
* **A rubric that passes trivially.** If every row passes at 100 no matter
  the input, the rubric is not discriminating — usually a `custom` prompt
  that grades "was an answer given" instead of a policy criterion. Rewrite
  the criterion against something the agent can actually violate.
* **Drift under traffic.** The dataset ages as your product changes: a
  baseline you froze six weeks ago no longer covers today's top contact
  reasons. Promote fresh failing conversations into the golden set
  regularly (everything in step 2) rather than growing a second dataset.
* **Compare rejects with `EVAL_COMPARE_DATASET_MISMATCH`.** The two runs
  were scored on different datasets — both must run against the same
  dataset id for a row-level verdict.
* **A `422` for `custom` with no prompt.** `rubric_prompt` is required when
  `rubric_name` is `custom`; the run is rejected at submit so a miswritten
  gate never silently runs.

## See also

* [Agent versions](/agents/agent-versions) — save the versions runs bind to.
* [Continuous production evals](/agents/continuous-production-evals) — keep
  scoring a deterministic slice of live traffic.
* [Simulation & Regression Eval Suite](/agents/simulation-eval-suite) —
  scripted scenario gate for multi-turn conversations.
* [Safely Roll Out an AI Agent](/guides/ai-agent-rollout-pipeline) — the
  canary ladder this gate feeds.
* [Agent evaluation endpoints](/api-reference/endpoints/agents) — the full
  route reference.