Fine-tune an agent end to end
An agent’s best conversations become its training data. The flow is: score transcripts with an eval run, export the best of them as provider-ready JSONL, run the tuning job on your own provider account, register the resulting model with Orbit, and deploy it to the agent. Each step is an API call; this page walks through all of them in order. Orbit orchestrates and tracks — your provider account does the training. Orbit never holds your provider credentials or submits a tuning job upstream. You upload the exported JSONL to OpenAI, Anthropic, or HuggingFace yourself, and record the provider’s job and model ids back on the Orbit job as it progresses.When fine-tuning beats prompting and RAG
Reach for fine-tuning when the gap is about how the agent responds, not what it knows:- Consistent tone and format at volume. If you keep correcting the same phrasing, structure, or greeting across thousands of conversations, distilling your best transcripts into the model removes that prompt weight and the drift that comes with it.
- A shorter, cheaper prompt. A tuned model can carry behaviors that would otherwise need a long system prompt — lower latency and lower per-run token cost.
- A stable house style. Format habits encoded in weights survive prompt edits and model upgrades better than instructions do.
- Facts that change. Product catalogs, policy documents, pricing — anything the agent must read fresh belongs in a knowledge base, retrieved by RAG. A fine-tuned model freezes whatever it absorbed at training time. See Knowledge base document inspector.
- Behavior instructions. If a prompt edit fixes the miss on your first try, keep it in the prompt. Weights are a commit; prompts are a draft.
- Thin datasets. An export below roughly a few hundred passing pairs rarely beats a good prompt. Filter to your best transcripts and check the line count before you spend on a training run.
Step 1 — Run an eval and export the dataset
Fine-tuning jobs anchor to a completed eval run, and the export uses that run’s scored rows as the training pairs. If you don’t have one yet, run your eval suite first — see Simulation and eval suite. Download a completed run’s judged transcripts as provider JSONL:
The response is a downloadable JSONL file, one training pair per line, capped at 1,000 examples. Exports default to judge-passed rows only — distill your best transcripts, not the average.
Step 2 — Run the tuning on your provider account
Upload the exported file to your provider’s fine-tuning surface (the OpenAI fine-tuning API, the Anthropic console, a HuggingFace training job) using your own account and credentials. Keep the job id the provider returns (ftjob-…-shaped on OpenAI) — you record it on the Orbit job in the next step.
Orbit’s position in this step is deliberate: the dataset never leaves through a Devotel-held provider key, the training bill lands on your provider account, and the resulting model id stays yours. Orbit tracks the lifecycle so the model is deployable once trained.
Step 3 — Create the managed job
Anchor an Orbit fine-tuning job to the eval run that produced the dataset:provider (openai | anthropic | hf), base_model, eval_run_id, and optionally format (default openai) and provider_job_id (if you already submitted upstream). The eval run must exist for this agent — otherwise the route returns 422 with error.code FINE_TUNE_EVAL_RUN_NOT_FOUND and a message telling you to export a completed eval run first. The job starts in queued.
Step 4 — Advance the lifecycle
Record progress as the external run moves. Allowed edges:queued → running, queued → cancelled, queued → failed; running → succeeded | failed | cancelled. Terminal states (succeeded, failed, cancelled) accept no further transitions — re-running tuning means creating a new job, so history is never rewritten.
succeeded transition without a fine_tuned_model returns 422 with error.code MISSING_FINE_TUNED_MODEL; an impossible edge returns 422 with INVALID_TRANSITION. Mark a failure the same way with status: "failed" plus an error string.
On success, the resulting model is registered into the agent’s custom-model registry automatically, and the response includes it as registered_model alongside the job. The job’s custom_model_id points at the registry entry.
Step 5 — Deploy to the agent
Point the agent at a registered model — this sets the agent’s active custom model, which the runtime resolves on the next run:409 with error.code FINE_TUNE_MODEL_NOT_DEPLOYED so a stale undeploy never drops a newer deploy:
Inspect the registry and jobs
One read returns the full fine-tuning state of an agent:jobs (with lifecycle status, provider job id, and error detail), custom_models (the registry — each entry holds provider, base_model, model_name, display_name, and the source_job_id it came from), and active_custom_model_id (null when the agent runs its base model). An agent holds up to 100 tracked jobs and 100 registered models; past that the writes return 422 with FINE_TUNE_JOBS_LIMIT or FINE_TUNE_MODELS_LIMIT.
The same state is visible in the dashboard on the agent’s Fine-tuning tab.
Where the state lives
Every step above reads and writes the agent’s ownconfig object — the same agent document you manage through the agents API. The surface adds three keys to that document: fine_tuning_jobs (tracked jobs), custom_models (the registry), and active_custom_model_id (the deploy pointer). There is no separate store: the registry travels with the agent, is covered by the same ownership and audit rules as the rest of agent config, and a deploy is just a flip of that pointer on the agent.
Writes require the agents:write scope and an owner, admin, or developer role; reads require agents:read. Every create, transition, deploy, and undeploy lands in the audit log.
See also
- Simulation and eval suite — build the dataset and run the evals that feed an export.
- Continuous production evals — keep scoring live traffic after you deploy a custom model.
- Agent model selection — how the base model is resolved before any custom model applies.
- Cost controls — cap spend per agent while you iterate on training runs.