> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Promote production traces into eval datasets

> Capture a real agent conversation from production and turn it into an eval dataset in one request — regression coverage built straight from live traffic.

# Promote production traces into eval datasets

The strongest eval rows come from real traffic. Instead of hand-authoring
every `{ input, expected_output }` row, promote a captured production
conversation into a new dataset in one request. This guide walks the full
capture workflow: read the trace, promote its turns, fix the failures, and
run the promoted dataset as a regression gate.

Endpoint paths below are relative. Send them against
`https://api.orbit.devotel.io/api/v1`. Promotion needs the `agents:write`
scope; reading the trace needs `agents:read`.

## 1. What promotion does

A promote call reads one production conversation's execution trace — the
same recorded turns the
[`GET /agents/{agentId}/conversations/{conversationId}/execution-trace`](/api-reference/endpoints/agents)
read surface returns — and creates a **new** eval dataset whose rows map
one-to-one from trace turns:

* **`input`** — the verbatim user prompt from that turn.
* **`expected_output`** — the assistant's response from that turn, as the
  baseline answer to grade against.
* **`metadata`** — provenance: the source conversation, the turn index, the
  turn's model, its audit outcome, when it was captured, and truncation
  flags if a long prompt or response had to be clipped.

That is a different source than hand-authoring rows (covered in
[Agent evals: datasets, runs, and pass-rate gates](/guides/agent-evals-datasets-loop))
and than the voice-only golden-set runner, which captures voice calls.
Promotion covers your text/chat agent traffic.

The promoted dataset is an ordinary dataset: it lists alongside scripted
datasets, its rows are editable, and the standard eval-run pipeline scores
it. Promotion is a creation path, not a special dataset type.

## 2. When to use it

Use promotion whenever live traffic surfaces a case your scripted rows
missed:

* **A failure you want as regression coverage.** A customer hit a bad
  answer in production; capture that turn as a row, fix the
  `expected_output`, and run it on every candidate so the agent can't
  regress into the same failure again.
* **Coverage drift.** Your golden set was authored weeks ago; promote the
  current top failure patterns rather than inventing hypothetical prompts.
* **Onboarding a new agent variant.** Seed a first dataset from a baseline
  agent's real conversations instead of writing case rows from scratch.

When the case is something you already know you want covered — a canonical
question, a policy rule, an edge case from a ticket — authoring the row by
hand is the right path. Promotion shines when the case came from traffic.

## 3. The input: a conversation's recorded turns

Promotion works on a conversation's recorded turn-by-turn surface: for
each turn, the user prompt the agent received, the assistant response it
gave, and the audit outcome recorded against that turn (`ok`, `refused`,
`error`, `escalated`, `cost_capped`).

The promote call maps that shape into rows server-side, in turn order:

* Turns missing a user prompt or an assistant response are **skipped** — a
  row needs both halves to be gradeable. The response tells you how many
  were skipped.
* Failure-outcome turns are **included by default** — capturing live
  failures is the point — and each row's `metadata.outcome` records the
  failure, so you know which rows need their expected answer corrected
  before the set is treated as golden. Pass
  `include_non_success_turns: false` to promote only known-good turns.

## 4. Promote a conversation

Send the conversation id to the promote route:

```bash theme={null}
curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/datasets/from-conversation \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "conversation_id": "conv_9f2a...",
    "name": "refund-failures-captured"
  }'
```

Body fields, all optional except `conversation_id`:

* `name` / `description` — defaults to "Promoted from \<conversation id>".
* `limit` — most-recent turn window, 1–200 (default 50).
* `include_non_success_turns` — defaults to `true`.

The response is `201` with the new dataset id plus the capture statistics:

```json theme={null}
{
  "data": {
    "id": "evalDs_8x1a",
    "agent_id": "agent_abc123",
    "name": "refund-failures-captured",
    "row_count": 12,
    "source_conversation_id": "conv_9f2a",
    "turns_considered": 14,
    "skipped_turns": 2,
    "capped_turns": 0
  }
}
```

Watch `skipped_turns`: rows are skipped when a turn lacked a user prompt or
an assistant response, or when you excluded non-success turns. A `422
EVAL_PROMOTE_NO_ROWS` means nothing was promotable — either the per-turn
audit log is disabled for your organization, or no turn carried both
halves. Response shape is on
[Promote a conversation trace into an eval dataset](/api-reference/endpoints/agents).

## 5. Worked example: capture, correct, run

**Find the conversation and inspect the trace.** Open the conversation in
the dashboard (conversations carry their id in the URL, `conv_...`), or
read the trace directly:

```bash theme={null}
curl "https://api.orbit.devotel.io/api/v1/agents/agent_abc123/conversations/conv_9f2a/execution-trace?limit=50" \
  -H "X-API-Key: dv_live_sk_..."
```

The trace returns ordered turns with model I/O, tool calls, outcomes, and
latency — check what the failing turn actually did before capturing it.

**Promote the turns you care about:**

```bash theme={null}
curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/datasets/from-conversation \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{"conversation_id": "conv_9f2a", "name": "q3-regression"}'
```

**Review and correct failure rows.** Read the dataset:

```bash theme={null}
curl "https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/datasets/evalDs_8x1a" \
  -H "X-API-Key: dv_live_sk_..."
```

Rows from failed turns carry `metadata.outcome != "ok"`; edit each one's
`expected_output` to the answer the agent **should** have given. Until you
do, the baseline is the wrong answer and the judge scores against it
verbatim.

**Run the dataset as a regression gate:**

```bash theme={null}
curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/runs \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{"dataset_id": "evalDs_8x1a", "rubric_name": "correctness", "threshold": 90}'
```

Poll `GET /agents/{agentId}/evals/runs/{runId}` until `completed`, then
read `passed_rows` / `total_rows`. Wire that pass rate into your rollout
gate exactly as you would for a scripted dataset — promotion only changes
where rows come from.

## 6. Limits and tips

* **One conversation per call.** Each promote call captures one trajectory;
  promote again for a second conversation.
* **1,000-row ceiling.** Datasets cap at 1,000 rows, same as the manual
  create route. Turns beyond the cap are dropped oldest-first (so the
  earliest turns win), with the count reported in `capped_turns`.
* **Row length ceiling.** Inputs and expected outputs over 10,000
  characters are clipped, with an `input_truncated` or
  `expected_output_truncated` flag set in row metadata.
* **Use `Idempotency-Key` when you retry.** Retry-safe deduplication works
  like every other write route; without a key, re-posting the same
  conversation creates a second dataset with a second id.
* **Where to find it.** In the dashboard: **Agents → your agent → Evals →
  Datasets**. Promote from the conversation's trace view; the promoted set
  appears in the same dataset list as scripted sets.
* **Correct the failures.** Rows from `error`/`refused`/`escalated`/
  `cost_capped` turns are captured deliberately — their whole value is that
  you fix the expected answer and turn a live failure into a gate.

## See also

* [Agent evals: datasets, runs, and pass-rate gates](/guides/agent-evals-datasets-loop) —
  the full eval loop this guide plugs into.
* [Agent conversation debug](/guides/agent-conversation-debug) — read a
  conversation's turns and tool calls before promoting.
* [Safely Roll Out an AI Agent](/guides/ai-agent-rollout-pipeline) — the
  canary gate your promoted datasets feed.
* [Agent evaluation endpoints](/api-reference/endpoints/agents) — the full
  route reference.