Agent evals: datasets, runs, and pass-rate gates

An eval run replays a fixed dataset of inputs against an agent version and lets an LLM judge score every response. Because the dataset and the rubric stay pinned, two runs compare cleanly — which is what makes a pass rate safe to gate a rollout on. This guide walks the full loop: declare a dataset, run it, triage the failures, and use the result as a gate. Endpoint paths below are relative. Send them against https://api.orbit.devotel.io/api/v1. All routes need agents:read / agents:write scope.

1. What an eval run expects

Every run binds three things, and the run row records all three so results stay attributable:

A dataset — up to 1,000 rows of { input, expected_output, metadata }. The runner sends each input to the agent and the judge compares the agent’s actual_output against your expected_output.
A rubric — one of correctness, helpfulness, safety, groundedness (faithfulness against retrieved knowledge-base context), or custom (your own rubric_prompt, required and up to 8,000 characters when you pick custom).
A judge model — the clause model that grades each row. Omit judge_model to inherit the default (claude-haiku-4-5-20251001); when you pin one it must be an approved Claude model id or the run is rejected at submit with a 422.

Runs always execute against a saved agent version. If you are testing a candidate, save it first — see Agent versions. For how production sampling keeps eval coverage on live traffic after launch, see Continuous production evals.

2. Declare a dataset

Create datasets from the Agents → your agent → Evals → Datasets page, or with POST /agents/{agentId}/evals/datasets. Two event sources feed a dataset; use both.

Scripted rows

Write rows by hand when you know the case you want covered — a canonical support question, a regression you just fixed, an edge case from a ticket.

curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/datasets \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "refund-golden-v1",
    "description": "Refund policy cases — quarter 3",
    "rows": [
      {
        "input": "You charged me twice for order 42. Refund the duplicate.",
        "expected_output": "Confirms the duplicate charge and initiates a refund with a clear timeline.",
        "metadata": { "source": "scripted", "case": "duplicate-charge" }
      }
    ]
  }'

The response is 201 with the dataset id and row_count. At least one row is required; the cap is 1,000 per dataset.

Promoted production conversations

The richer source is live traffic. One turn of a captured conversation’s execution trace becomes one { input: user prompt, expected_output: assistant response } row:

curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/datasets/from-conversation \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "conversation_id": "conv_9f2a...",
    "name": "refund-failures-captured",
    "limit": 50
  }'

include_non_success_turns defaults to true — on purpose, because capturing the turns where the agent’s audit outcome was a failure is exactly what you want in the golden set. Narrow the window with limit (defaults to the most recent turns, max 200). Read back what you have with GET /agents/{agentId}/evals/datasets (list) and GET /agents/{agentId}/evals/datasets/{dsId} (rows). Dataset names and rows are editable later — rename a set, fix a mislabeled expected_output, or prune a stale case.

3. Run a suite

Start a run from the Evals → Runs page or with POST /agents/{agentId}/evals/runs:

curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/runs \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_id": "evalDs_abc",
    "rubric_name": "correctness",
    "threshold": 80
  }'

The response is 202 with status: "pending" — the run executes in the background. Poll GET /agents/{agentId}/evals/runs/{runId} until the status flips to completed (or failed), then read the aggregate passed_rows, failed_rows, and avg_score on the same object. GET /agents/{agentId}/evals/runs lists recent runs; filter with ?status=completed. Pick a rubric that matches the failure you are guarding against. For a knowledge-grounded agent, groundedness is the stronger check — it catches answers that read well but are not in the retrieval context.

4. Read the failures

The run detail returns a results array (up to 1,000 rows) with, per row: actual_output, the judge’s score, a boolean passed, and judge_reasoning, plus latency_ms and cost_cents. Triage in this order:

Filter to passed: false. Each failed row carries the exact dataset_row_id, so you always know which input regressed.
Read judge_reasoning before touching the prompt. The judge names which rubric item failed — “no refund timeline given” is actionable; “score 62” alone is not.
Check error_message. A row that errored (runtime fault, not a judge reject) shows the cause in the row itself; fix the fault before treating it as a quality regression.
Compare against the baseline run. GET /agents/{agentId}/evals/runs/compare?run_a_id=<old>&run_b_id=<new> reports the same verdict on the same rows — it rejects with a 400 EVAL_COMPARE_DATASET_MISMATCH if the two runs were not run against the same dataset.

For knowledge-grounded failures, the route to fix is the knowledge source, not the prompt — the retrieval benchmark in the rollout pipeline guide isolates that.

5. Gate on a pass-rate threshold

Each run carries a threshold (0–100, default 70): the judge score a row must reach to count as passed. Set it at submit time and keep it pinned across runs so pass rates move only because the agent moved. Wire the gate like this:

Run the pinned dataset against the candidate version.
Read passed_rows / total_rows; require the pass rate to clear your gate (for example, all rows passed, or at least 90% on a noisy rubric).
Promote only then — via the canary ladder in the rollout pipeline guide, never by hand.

A CI job does the same thing mechanically: poll GET /agents/{agentId}/evals/runs/{runId} until completed, then assert passed_rows / total_rows clears the gate before promoting. The threshold stays a row-level judge cutoff; the gate asserts on the aggregate.

6. Running the eval loop from code

Every step above is scriptable — the same CI gate described in step 5 is a sequence of five calls. Each step below ships cURL, TypeScript, and Python side by side, matching the language floor in How request samples work. The TypeScript form uses the SDK’s untyped request method (the typed resources do not yet expose eval helpers); the Python form uses plain requests.

1 — Create the dataset

curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/datasets \
  -H "X-API-Key: $ORBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "refund-golden-v1",
    "description": "Refund policy cases — quarter 3",
    "rows": [
      {
        "input": "You charged me twice for order 42. Refund the duplicate.",
        "expected_output": "Confirms the duplicate charge and initiates a refund with a clear timeline.",
        "metadata": { "source": "scripted", "case": "duplicate-charge" }
      }
    ]
  }'

import { Orbit } from '@devotel-orbit/node'

const orbit = new Orbit({ apiKey: process.env.ORBIT_API_KEY })

const created = await orbit.request(
  'POST',
  '/agents/agent_abc123/evals/datasets',
  {
    name: 'refund-golden-v1',
    description: 'Refund policy cases — quarter 3',
    rows: [
      {
        input: 'You charged me twice for order 42. Refund the duplicate.',
        expected_output:
          'Confirms the duplicate charge and initiates a refund with a clear timeline.',
        metadata: { source: 'scripted', case: 'duplicate-charge' },
      },
    ],
  },
)

const datasetId = created.data.id

import os, requests

headers = {
    "X-API-Key": os.environ["ORBIT_API_KEY"],
    "Content-Type": "application/json",
}

payload = {
    "name": "refund-golden-v1",
    "description": "Refund policy cases — quarter 3",
    "rows": [
        {
            "input": "You charged me twice for order 42. Refund the duplicate.",
            "expected_output": "Confirms the duplicate charge and initiates a refund with a clear timeline.",
            "metadata": {"source": "scripted", "case": "duplicate-charge"},
        }
    ],
}

r = requests.post(
    "https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/datasets",
    headers=headers,
    json=payload,
)
r.raise_for_status()
dataset_id = r.json()["data"]["id"]

The response is 201 Created — the full envelope, data plus meta:

201 Created

{
  "data": {
    "id": "evalDs_abc",
    "agent_id": "agent_abc123",
    "name": "refund-golden-v1",
    "row_count": 1
  },
  "meta": { "request_id": "req_01J9", "timestamp": "2026-09-28T09:12:04.418Z" }
}

2 — Import a conversation

Enrich the same dataset with turns captured from production traffic:

curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/datasets/from-conversation \
  -H "X-API-Key: $ORBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "conversation_id": "conv_9f2a...",
    "name": "refund-golden-v1",
    "limit": 50
  }'

await orbit.request(
  'POST',
  '/agents/agent_abc123/evals/datasets/from-conversation',
  {
    conversation_id: 'conv_9f2a...',
    name: 'refund-golden-v1',
    limit: 50,
  },
)

r = requests.post(
    "https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/datasets/from-conversation",
    headers=headers,
    json={
        "conversation_id": "conv_9f2a...",
        "name": "refund-golden-v1",
        "limit": 50,
    },
)
r.raise_for_status()

3 — Submit the run

curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/runs \
  -H "X-API-Key: $ORBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_id": "evalDs_abc",
    "rubric_name": "correctness",
    "threshold": 80
  }'

const started = await orbit.request('POST', '/agents/agent_abc123/evals/runs', {
  dataset_id: 'evalDs_abc',
  rubric_name: 'correctness',
  threshold: 80,
})

const runId = started.data.id

r = requests.post(
    "https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/runs",
    headers=headers,
    json={
        "dataset_id": "evalDs_abc",
        "rubric_name": "correctness",
        "threshold": 80,
    },
)
r.raise_for_status()
run_id = r.json()["data"]["id"]

The submit returns 202 Accepted with status: "pending" — the run executes in the background and you poll for the terminal state:

202 Accepted

{
  "data": {
    "id": "evalRun_xyz",
    "agent_id": "agent_abc123",
    "dataset_id": "evalDs_abc",
    "rubric_name": "correctness",
    "judge_model": "claude-haiku-4-5-20251001",
    "threshold": 80,
    "status": "pending"
  },
  "meta": { "request_id": "req_01JA", "timestamp": "2026-09-28T09:12:07.203Z" }
}

4 — Poll until the run completes

Poll GET /agents/{agentId}/evals/runs/{runId} until status flips to completed (or failed), then read the aggregate passed_rows / failed_rows that the gate asserts on:

curl https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/runs/evalRun_xyz \
  -H "X-API-Key: $ORBIT_API_KEY"

async function waitForRun(runId: string) {
  for (;;) {
    const res = await orbit.request(
      'GET',
      `/agents/agent_abc123/evals/runs/${runId}`,
    )
    const run = res.data
    if (run.status === 'completed' || run.status === 'failed') return run
    await new Promise((r) => setTimeout(r, 5000))
  }
}

const run = await waitForRun(runId)
if (run.status !== 'completed') throw new Error(run.error_message)

const passRate = run.passed_rows / run.total_rows
if (passRate < 0.9) {
  throw new Error(
    `Gate failed: ${run.passed_rows}/${run.total_rows} passed (${run.failed_rows} failed, avg_score ${run.avg_score})`,
  )
}

import time

def wait_for_run(run_id):
    while True:
        r = requests.get(
            f"https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/runs/{run_id}",
            headers=headers,
        )
        r.raise_for_status()
        run = r.json()["data"]
        if run["status"] in ("completed", "failed"):
            return run
        time.sleep(5)

run = wait_for_run(run_id)
if run["status"] != "completed":
    raise SystemExit(f"Run failed: {run['error_message']}")

pass_rate = run["passed_rows"] / run["total_rows"]
if pass_rate < 0.9:
    raise SystemExit(
        f"Gate failed: {run['passed_rows']}/{run['total_rows']} passed "
        f"({run['failed_rows']} failed, avg_score {run['avg_score']})"
    )

5 — Compare runs

Score the candidate against the frozen baseline (step 6), then gate on the verdict:

curl "https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/runs/compare?run_a_id=evalRun_base&run_b_id=evalRun_xyz" \
  -H "X-API-Key: $ORBIT_API_KEY"

const verdict = await orbit.request(
  'GET',
  '/agents/agent_abc123/evals/runs/compare?run_a_id=evalRun_base&run_b_id=evalRun_xyz',
)

r = requests.get(
    "https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/runs/compare",
    headers=headers,
    params={"run_a_id": "evalRun_base", "run_b_id": "evalRun_xyz"},
)
r.raise_for_status()
verdict = r.json()["data"]

Compare only accepts two runs that were scored on the same dataset. Feed it runs from different datasets and it rejects with 400 — the error envelope matches the shape in Error codes:

400 Error

{
  "error": {
    "code": "EVAL_COMPARE_DATASET_MISMATCH",
    "message": "Both runs must reference the same dataset to be compared side-by-side",
    "status": 400
  },
  "meta": { "request_id": "req_01JB", "timestamp": "2026-09-28T09:12:19.811Z" }
}

Re-run the baseline against the current dataset and compare again.

7. Holdout vs experiment

Keep a holdout when you experiment.

Freeze a baseline. Save the current production prompt as a version and never edit it. Run the pinned dataset against that version once and keep the run id as your reference.
Score experiments against the holdout. An experiment splits live traffic between the control version (variant A) and the candidate (variant B). Run the same dataset against both versions and use GET /agents/{agentId}/evals/runs/compare for the verdict — never compare a candidate against a dataset that drifted since the baseline was scored.
Delete stales. A baseline run on a dataset you have since edited is a false holdout. Re-run the baseline after any dataset change, and record both run ids on the experiment.

8. Failure modes

A run stuck at pending. The background runner never picked it up — re-submit and check GET /agents/{agentId}/evals/runs?status=failed for an error_message on the sibling rows. A run that never terminates almost always carries its cause on the failed run detail.
A rubric that passes trivially. If every row passes at 100 no matter the input, the rubric is not discriminating — usually a custom prompt that grades “was an answer given” instead of a policy criterion. Rewrite the criterion against something the agent can actually violate.
Drift under traffic. The dataset ages as your product changes: a baseline you froze six weeks ago no longer covers today’s top contact reasons. Promote fresh failing conversations into the golden set regularly (everything in step 2) rather than growing a second dataset.
Compare rejects with EVAL_COMPARE_DATASET_MISMATCH. The two runs were scored on different datasets — both must run against the same dataset id for a row-level verdict.
A 422 for custom with no prompt. rubric_prompt is required when rubric_name is custom; the run is rejected at submit so a miswritten gate never silently runs.

WhatsApp

Developer console

Contact center

Guides

Messaging Controls

Audience & Support Tools

Agent evals: datasets, runs, and pass-rate gates

Agent evals: datasets, runs, and pass-rate gates

1. What an eval run expects

2. Declare a dataset

Scripted rows

Promoted production conversations

3. Run a suite

4. Read the failures

5. Gate on a pass-rate threshold

6. Running the eval loop from code

1 — Create the dataset

2 — Import a conversation

3 — Submit the run

4 — Poll until the run completes

5 — Compare runs

7. Holdout vs experiment

8. Failure modes

See also

​Agent evals: datasets, runs, and pass-rate gates

​1. What an eval run expects

​2. Declare a dataset

​Scripted rows

​Promoted production conversations

​3. Run a suite

​4. Read the failures

​5. Gate on a pass-rate threshold

​6. Running the eval loop from code

​1 — Create the dataset

​2 — Import a conversation

​3 — Submit the run

​4 — Poll until the run completes

​5 — Compare runs

​7. Holdout vs experiment

​8. Failure modes

​See also

Agent evals: datasets, runs, and pass-rate gates

1. What an eval run expects

2. Declare a dataset

Scripted rows

Promoted production conversations

3. Run a suite

4. Read the failures

5. Gate on a pass-rate threshold

6. Running the eval loop from code

1 — Create the dataset

2 — Import a conversation

3 — Submit the run

4 — Poll until the run completes

5 — Compare runs

7. Holdout vs experiment

8. Failure modes

See also