> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Fine-tuning: dataset export to deployed custom model

> The end-to-end fine-tuning path: export a judged eval dataset as provider JSONL, track the tuning job, register the resulting model in the agent registry, and deploy it.

# Fine-tune an agent end to end

An agent's best conversations become its training data. The flow is: score transcripts with an eval run, export the best of them as provider-ready JSONL, run the tuning job on your own provider account, register the resulting model with Orbit, and deploy it to the agent. Each step is an API call; this page walks through all of them in order.

Orbit orchestrates and tracks — your provider account does the training. Orbit never holds your provider credentials or submits a tuning job upstream. You upload the exported JSONL to OpenAI, Anthropic, or HuggingFace yourself, and record the provider's job and model ids back on the Orbit job as it progresses.

## When fine-tuning beats prompting and RAG

Reach for fine-tuning when the gap is about *how* the agent responds, not *what* it knows:

* **Consistent tone and format at volume.** If you keep correcting the same phrasing, structure, or greeting across thousands of conversations, distilling your best transcripts into the model removes that prompt weight and the drift that comes with it.
* **A shorter, cheaper prompt.** A tuned model can carry behaviors that would otherwise need a long system prompt — lower latency and lower per-run token cost.
* **A stable house style.** Format habits encoded in weights survive prompt edits and model upgrades better than instructions do.

Fine-tuning is the wrong tool when the gap is knowledge:

* **Facts that change.** Product catalogs, policy documents, pricing — anything the agent must read fresh belongs in a knowledge base, retrieved by RAG. A fine-tuned model freezes whatever it absorbed at training time. See [Knowledge base document inspector](/agents/kb-document-inspector).
* **Behavior instructions.** If a prompt edit fixes the miss on your first try, keep it in the prompt. Weights are a commit; prompts are a draft.
* **Thin datasets.** An export below roughly a few hundred passing pairs rarely beats a good prompt. Filter to your best transcripts and check the line count before you spend on a training run.

## Step 1 — Run an eval and export the dataset

Fine-tuning jobs anchor to a completed eval run, and the export uses that run's scored rows as the training pairs. If you don't have one yet, run your eval suite first — see [Simulation and eval suite](/agents/simulation-eval-suite).

Download a completed run's judged transcripts as provider JSONL:

```bash theme={null}
curl "https://api.orbit.devotel.io/api/v1/agents/agent_abc123/evals/runs/run_xyz789/export?format=openai" \
  -H "X-API-Key: dv_live_sk_..." \
  -o training-data.jsonl
```

Query parameters:

| Parameter           | Values                      | Default  | Meaning                                                                                                                                        |
| ------------------- | --------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `format` (required) | `openai`, `anthropic`, `hf` | —        | Provider JSONL shape. OpenAI uses a `messages` array, Anthropic puts `system` at the top level, HuggingFace emits `prompt`/`completion` pairs. |
| `source`            | `actual`, `expected`        | `actual` | Completion side of each pair: the agent's own judged response, or the dataset's golden reference answer.                                       |
| `passed_only`       | `true`, `false`             | `true`   | Restrict to rows the judge passed. Off means you also export failures — rarely what you want to train on.                                      |
| `min_score`         | `0`–`100`                   | —        | Score floor applied on top of `passed_only`.                                                                                                   |
| `system_prompt`     | string                      | —        | System prompt embedded into every example (`openai` and `anthropic` formats only).                                                             |

The response is a downloadable JSONL file, one training pair per line, capped at 1,000 examples. Exports default to judge-passed rows only — distill your best transcripts, not the average.

## Step 2 — Run the tuning on your provider account

Upload the exported file to your provider's fine-tuning surface (the OpenAI fine-tuning API, the Anthropic console, a HuggingFace training job) using your own account and credentials. Keep the job id the provider returns (`ftjob-…`-shaped on OpenAI) — you record it on the Orbit job in the next step.

Orbit's position in this step is deliberate: the dataset never leaves through a Devotel-held provider key, the training bill lands on your provider account, and the resulting model id stays yours. Orbit tracks the lifecycle so the model is deployable once trained.

## Step 3 — Create the managed job

Anchor an Orbit fine-tuning job to the eval run that produced the dataset:

```bash theme={null}
curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/fine-tuning/jobs \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "provider": "openai",
    "base_model": "gpt-4o-mini-2024-07-18",
    "eval_run_id": "run_xyz789",
    "format": "openai",
    "provider_job_id": "ftjob-abc123"
  }'
```

The body takes `provider` (`openai` | `anthropic` | `hf`), `base_model`, `eval_run_id`, and optionally `format` (default `openai`) and `provider_job_id` (if you already submitted upstream). The eval run must exist for this agent — otherwise the route returns `422` with `error.code` `FINE_TUNE_EVAL_RUN_NOT_FOUND` and a message telling you to export a completed eval run first. The job starts in `queued`.

## Step 4 — Advance the lifecycle

Record progress as the external run moves. Allowed edges: `queued → running`, `queued → cancelled`, `queued → failed`; `running → succeeded | failed | cancelled`. Terminal states (`succeeded`, `failed`, `cancelled`) accept no further transitions — re-running tuning means creating a new job, so history is never rewritten.

```bash theme={null}
curl -X PATCH https://api.orbit.devotel.io/api/v1/agents/agent_abc123/fine-tuning/jobs/agentFineTuneJob_def456 \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{ "status": "running", "provider_job_id": "ftjob-abc123" }'
```

When the provider finishes, mark the job succeeded and name the resulting model:

```bash theme={null}
curl -X PATCH https://api.orbit.devotel.io/api/v1/agents/agent_abc123/fine-tuning/jobs/agentFineTuneJob_def456 \
  -H "X-API-Key: dv_live_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "status": "succeeded",
    "fine_tuned_model": "ft:gpt-4o-mini:acme::greeting-v3",
    "model_display_name": "Support tone v3"
  }'
```

A `succeeded` transition without a `fine_tuned_model` returns `422` with `error.code` `MISSING_FINE_TUNED_MODEL`; an impossible edge returns `422` with `INVALID_TRANSITION`. Mark a failure the same way with `status: "failed"` plus an `error` string.

On success, the resulting model is registered into the agent's custom-model registry automatically, and the response includes it as `registered_model` alongside the job. The job's `custom_model_id` points at the registry entry.

## Step 5 — Deploy to the agent

Point the agent at a registered model — this sets the agent's active custom model, which the runtime resolves on the next run:

```bash theme={null}
curl -X POST https://api.orbit.devotel.io/api/v1/agents/agent_abc123/fine-tuning/models/agentCustomModel_ghi789/deploy \
  -H "X-API-Key: dv_live_sk_..."
```

Revert the agent to its base model by undeploying. Only the *currently active* model can be undeployed — a mismatch returns `409` with `error.code` `FINE_TUNE_MODEL_NOT_DEPLOYED` so a stale undeploy never drops a newer deploy:

```bash theme={null}
curl -X DELETE https://api.orbit.devotel.io/api/v1/agents/agent_abc123/fine-tuning/models/agentCustomModel_ghi789/deploy \
  -H "X-API-Key: dv_live_sk_..."
```

## Inspect the registry and jobs

One read returns the full fine-tuning state of an agent:

```bash theme={null}
curl https://api.orbit.devotel.io/api/v1/agents/agent_abc123/fine-tuning/jobs \
  -H "X-API-Key: dv_live_sk_..."
```

The response carries `jobs` (with lifecycle status, provider job id, and error detail), `custom_models` (the registry — each entry holds `provider`, `base_model`, `model_name`, `display_name`, and the `source_job_id` it came from), and `active_custom_model_id` (`null` when the agent runs its base model). An agent holds up to 100 tracked jobs and 100 registered models; past that the writes return `422` with `FINE_TUNE_JOBS_LIMIT` or `FINE_TUNE_MODELS_LIMIT`.

The same state is visible in the dashboard on the agent's **Fine-tuning** tab.

## Where the state lives

Every step above reads and writes the agent's own `config` object — the same agent document you manage through the agents API. The surface adds three keys to that document: `fine_tuning_jobs` (tracked jobs), `custom_models` (the registry), and `active_custom_model_id` (the deploy pointer). There is no separate store: the registry travels with the agent, is covered by the same ownership and audit rules as the rest of agent config, and a deploy is just a flip of that pointer on the agent.

Writes require the `agents:write` scope and an owner, admin, or developer role; reads require `agents:read`. Every create, transition, deploy, and undeploy lands in the audit log.

## See also

* [Simulation and eval suite](/agents/simulation-eval-suite) — build the dataset and run the evals that feed an export.
* [Continuous production evals](/agents/continuous-production-evals) — keep scoring live traffic after you deploy a custom model.
* [Agent model selection](/agents/model-selection) — how the base model is resolved before any custom model applies.
* [Cost controls](/agents/cost-controls) — cap spend per agent while you iterate on training runs.