Skip to main content

​
Promote production traces into eval datasets

The strongest eval rows come from real traffic. Instead of hand-authoring every { input, expected_output } row, promote a captured production conversation into a new dataset in one request. This guide walks the full capture workflow: read the trace, promote its turns, fix the failures, and run the promoted dataset as a regression gate. Endpoint paths below are relative. Send them against https://api.orbit.devotel.io/api/v1. Promotion needs the agents:write scope; reading the trace needs agents:read.

​
1. What promotion does

A promote call reads one production conversation’s execution trace — the same recorded turns the GET /agents/{agentId}/conversations/{conversationId}/execution-trace read surface returns — and creates a new eval dataset whose rows map one-to-one from trace turns:
  • input — the verbatim user prompt from that turn.
  • expected_output — the assistant’s response from that turn, as the baseline answer to grade against.
  • metadata — provenance: the source conversation, the turn index, the turn’s model, its audit outcome, when it was captured, and truncation flags if a long prompt or response had to be clipped.
That is a different source than hand-authoring rows (covered in Agent evals: datasets, runs, and pass-rate gates) and than the voice-only golden-set runner, which captures voice calls. Promotion covers your text/chat agent traffic. The promoted dataset is an ordinary dataset: it lists alongside scripted datasets, its rows are editable, and the standard eval-run pipeline scores it. Promotion is a creation path, not a special dataset type.

​
2. When to use it

Use promotion whenever live traffic surfaces a case your scripted rows missed:
  • A failure you want as regression coverage. A customer hit a bad answer in production; capture that turn as a row, fix the expected_output, and run it on every candidate so the agent can’t regress into the same failure again.
  • Coverage drift. Your golden set was authored weeks ago; promote the current top failure patterns rather than inventing hypothetical prompts.
  • Onboarding a new agent variant. Seed a first dataset from a baseline agent’s real conversations instead of writing case rows from scratch.
When the case is something you already know you want covered — a canonical question, a policy rule, an edge case from a ticket — authoring the row by hand is the right path. Promotion shines when the case came from traffic.

​
3. The input: a conversation’s recorded turns

Promotion works on a conversation’s recorded turn-by-turn surface: for each turn, the user prompt the agent received, the assistant response it gave, and the audit outcome recorded against that turn (ok, refused, error, escalated, cost_capped). The promote call maps that shape into rows server-side, in turn order:
  • Turns missing a user prompt or an assistant response are skipped — a row needs both halves to be gradeable. The response tells you how many were skipped.
  • Failure-outcome turns are included by default — capturing live failures is the point — and each row’s metadata.outcome records the failure, so you know which rows need their expected answer corrected before the set is treated as golden. Pass include_non_success_turns: false to promote only known-good turns.

​
4. Promote a conversation

Send the conversation id to the promote route:
Body fields, all optional except conversation_id:
  • name / description — defaults to “Promoted from <conversation id>”.
  • limit — most-recent turn window, 1–200 (default 50).
  • include_non_success_turns — defaults to true.
The response is 201 with the new dataset id plus the capture statistics:
Watch skipped_turns: rows are skipped when a turn lacked a user prompt or an assistant response, or when you excluded non-success turns. A 422 EVAL_PROMOTE_NO_ROWS means nothing was promotable — either the per-turn audit log is disabled for your organization, or no turn carried both halves. Response shape is on Promote a conversation trace into an eval dataset.

​
5. Worked example: capture, correct, run

Find the conversation and inspect the trace. Open the conversation in the dashboard (conversations carry their id in the URL, conv_...), or read the trace directly:
The trace returns ordered turns with model I/O, tool calls, outcomes, and latency — check what the failing turn actually did before capturing it. Promote the turns you care about:
Review and correct failure rows. Read the dataset:
Rows from failed turns carry metadata.outcome != "ok"; edit each one’s expected_output to the answer the agent should have given. Until you do, the baseline is the wrong answer and the judge scores against it verbatim. Run the dataset as a regression gate:
Poll GET /agents/{agentId}/evals/runs/{runId} until completed, then read passed_rows / total_rows. Wire that pass rate into your rollout gate exactly as you would for a scripted dataset — promotion only changes where rows come from.

​
6. Limits and tips

  • One conversation per call. Each promote call captures one trajectory; promote again for a second conversation.
  • 1,000-row ceiling. Datasets cap at 1,000 rows, same as the manual create route. Turns beyond the cap are dropped oldest-first (so the earliest turns win), with the count reported in capped_turns.
  • Row length ceiling. Inputs and expected outputs over 10,000 characters are clipped, with an input_truncated or expected_output_truncated flag set in row metadata.
  • Use Idempotency-Key when you retry. Retry-safe deduplication works like every other write route; without a key, re-posting the same conversation creates a second dataset with a second id.
  • Where to find it. In the dashboard: Agents → your agent → Evals → Datasets. Promote from the conversation’s trace view; the promoted set appears in the same dataset list as scripted sets.
  • Correct the failures. Rows from error/refused/escalated/ cost_capped turns are captured deliberately — their whole value is that you fix the expected answer and turn a live failure into a gate.

​
See also