Skip to main content

​
Agent evals: datasets, runs, and pass-rate gates

An eval run replays a fixed dataset of inputs against an agent version and lets an LLM judge score every response. Because the dataset and the rubric stay pinned, two runs compare cleanly — which is what makes a pass rate safe to gate a rollout on. This guide walks the full loop: declare a dataset, run it, triage the failures, and use the result as a gate. Endpoint paths below are relative. Send them against https://api.orbit.devotel.io/api/v1. All routes need agents:read / agents:write scope.

​
1. What an eval run expects

Every run binds three things, and the run row records all three so results stay attributable:
  • A dataset — up to 1,000 rows of { input, expected_output, metadata }. The runner sends each input to the agent and the judge compares the agent’s actual_output against your expected_output.
  • A rubric — one of correctness, helpfulness, safety, groundedness (faithfulness against retrieved knowledge-base context), or custom (your own rubric_prompt, required and up to 8,000 characters when you pick custom).
  • A judge model — the clause model that grades each row. Omit judge_model to inherit the default (claude-haiku-4-5-20251001); when you pin one it must be an approved Claude model id or the run is rejected at submit with a 422.
Runs always execute against a saved agent version. If you are testing a candidate, save it first — see Agent versions. For how production sampling keeps eval coverage on live traffic after launch, see Continuous production evals.

​
2. Declare a dataset

Create datasets from the Agents → your agent → Evals → Datasets page, or with POST /agents/{agentId}/evals/datasets. Two event sources feed a dataset; use both.

​
Scripted rows

Write rows by hand when you know the case you want covered — a canonical support question, a regression you just fixed, an edge case from a ticket.
The response is 201 with the dataset id and row_count. At least one row is required; the cap is 1,000 per dataset. The richer source is live traffic. One turn of a captured conversation’s execution trace becomes one { input: user prompt, expected_output: assistant response } row:
include_non_success_turns defaults to true — on purpose, because capturing the turns where the agent’s audit outcome was a failure is exactly what you want in the golden set. Narrow the window with limit (defaults to the most recent turns, max 200). Read back what you have with GET /agents/{agentId}/evals/datasets (list) and GET /agents/{agentId}/evals/datasets/{dsId} (rows). Dataset names and rows are editable later — rename a set, fix a mislabeled expected_output, or prune a stale case.

​
3. Run a suite

Start a run from the Evals → Runs page or with POST /agents/{agentId}/evals/runs:
The response is 202 with status: "pending" — the run executes in the background. Poll GET /agents/{agentId}/evals/runs/{runId} until the status flips to completed (or failed), then read the aggregate passed_rows, failed_rows, and avg_score on the same object. GET /agents/{agentId}/evals/runs lists recent runs; filter with ?status=completed. Pick a rubric that matches the failure you are guarding against. For a knowledge-grounded agent, groundedness is the stronger check — it catches answers that read well but are not in the retrieval context.

​
4. Read the failures

The run detail returns a results array (up to 1,000 rows) with, per row: actual_output, the judge’s score, a boolean passed, and judge_reasoning, plus latency_ms and cost_cents. Triage in this order:
  1. Filter to passed: false. Each failed row carries the exact dataset_row_id, so you always know which input regressed.
  2. Read judge_reasoning before touching the prompt. The judge names which rubric item failed — “no refund timeline given” is actionable; “score 62” alone is not.
  3. Check error_message. A row that errored (runtime fault, not a judge reject) shows the cause in the row itself; fix the fault before treating it as a quality regression.
  4. Compare against the baseline run. GET /agents/{agentId}/evals/runs/compare?run_a_id=<old>&run_b_id=<new> reports the same verdict on the same rows — it rejects with a 400 EVAL_COMPARE_DATASET_MISMATCH if the two runs were not run against the same dataset.
For knowledge-grounded failures, the route to fix is the knowledge source, not the prompt — the retrieval benchmark in the rollout pipeline guide isolates that.

​
5. Gate on a pass-rate threshold

Each run carries a threshold (0–100, default 70): the judge score a row must reach to count as passed. Set it at submit time and keep it pinned across runs so pass rates move only because the agent moved. Wire the gate like this:
  1. Run the pinned dataset against the candidate version.
  2. Read passed_rows / total_rows; require the pass rate to clear your gate (for example, all rows passed, or at least 90% on a noisy rubric).
  3. Promote only then — via the canary ladder in the rollout pipeline guide, never by hand.
A CI job does the same thing mechanically: poll GET /agents/{agentId}/evals/runs/{runId} until completed, then assert passed_rows / total_rows clears the gate before promoting. The threshold stays a row-level judge cutoff; the gate asserts on the aggregate.

​
6. Holdout vs experiment

Keep a holdout when you experiment.
  • Freeze a baseline. Save the current production prompt as a version and never edit it. Run the pinned dataset against that version once and keep the run id as your reference.
  • Score experiments against the holdout. An experiment splits live traffic between the control version (variant A) and the candidate (variant B). Run the same dataset against both versions and use GET /agents/{agentId}/evals/runs/compare for the verdict — never compare a candidate against a dataset that drifted since the baseline was scored.
  • Delete stales. A baseline run on a dataset you have since edited is a false holdout. Re-run the baseline after any dataset change, and record both run ids on the experiment.

​
7. Failure modes

  • A run stuck at pending. The background runner never picked it up — re-submit and check GET /agents/{agentId}/evals/runs?status=failed for an error_message on the sibling rows. A run that never terminates almost always carries its cause on the failed run detail.
  • A rubric that passes trivially. If every row passes at 100 no matter the input, the rubric is not discriminating — usually a custom prompt that grades “was an answer given” instead of a policy criterion. Rewrite the criterion against something the agent can actually violate.
  • Drift under traffic. The dataset ages as your product changes: a baseline you froze six weeks ago no longer covers today’s top contact reasons. Promote fresh failing conversations into the golden set regularly (everything in step 2) rather than growing a second dataset.
  • Compare rejects with EVAL_COMPARE_DATASET_MISMATCH. The two runs were scored on different datasets — both must run against the same dataset id for a row-level verdict.
  • A 422 for custom with no prompt. rubric_prompt is required when rubric_name is custom; the run is rejected at submit so a miswritten gate never silently runs.

​
See also