Skip to main content

​
Async batch inference for AI agents

The batch inference API lets you submit up to 500 agent invocations in one call and collect the results asynchronously. It exists for workloads where a synchronous /chat turn is the wrong shape: anything where you have a pile of inputs, you want the agent’s judgment on each one, and you don’t need an answer inside the request/response. Typical workloads:
  • Mass classification — label 400 inbound messages by intent or sentiment for a routing model.
  • Summarization — condense yesterday’s resolved conversations into nightly digests.
  • Data labeling — generate training pairs for an internal model.
  • CDP enrichment — extract a preference or attribute from each contact’s latest thread and write it back to the contact record.
Use live /chat when a person or a UI is waiting on the reply. Use the batch when throughput matters and latency doesn’t.

​
How a batch works

A batch is submitted with one POST, processed item by item by a background worker, and retained so you can collect results when it finishes:
  1. Submit — POST /agents/:id/batch validates and queues the batch, and answers 202 Accepted with a batch_id immediately.
  2. Track — GET /agents/:id/batch/:batchId returns the live status (queued → processing → completed/failed) plus progress counts while the worker drains the queue.
  3. Collect — when the batch completes, either poll the same GET or subscribe to the agent.batch.completed tenant webhook, then fetch the full per-item result set.
Submit and read require owner, admin, or developer roles and the agents:write / agents:read scopes respectively.

​
Submit a batch

POST /api/v1/agents/:id/batch accepts up to 500 items. Each item carries the message to send the agent, an optional prior-turn history (up to 20 turns) so an item can continue a thread, and an optional custom_id (up to 120 chars) that is echoed back on the result so you can join the outcome to your own records. Any item without a custom_id gets a generated one. custom_id must be unique within the batch — a duplicate is rejected with 422 BATCH_DUPLICATE_CUSTOM_ID so the result set stays unambiguous.
The response is 202 Accepted — the batch is queued, not started:
Error surface at submit:
  • 404 NOT_FOUND — the agent does not exist in your tenant; the batch is not queued.
  • 422 BATCH_DUPLICATE_CUSTOM_ID — an item repeats a custom_id; resubmit with unique ids.
  • 422 VALIDATION_ERROR — the body failed validation (over 500 items, an input over 8000 characters, more than 20 history turns).
  • 503 BATCH_QUEUE_UNAVAILABLE — the batch queue is temporarily unavailable; nothing was accepted, so retry shortly.
To pin every item to a specific published version of the agent, pass agent_version_id at the top level. Omit it and the batch runs against the agent’s current candidate version.

​
Poll for status and results

GET /api/v1/agents/:id/batch/:batchId reports the batch’s current state. While the worker is draining the queue you get live progress; when it finishes the same response carries the aggregate summary and the full per-item result set.
In flight:
Completed — status flips to completed, summary rolls up the roll-call, and results holds one entry per submitted item in submission order:
A per-item failure never sinks the batch: failed items are recorded with an error message and zero usage, and the batch still completes. Only an infrastructure-level failure flips the whole batch to failed, in which case error carries the job-level reason. Tenant isolation: a batch id resolves only for the tenant and agent that submitted it. A valid id polled by another tenant, or under another agent, returns 404 — never the other tenant’s results.

​
The completion webhook

Polling works, but the better pattern is to subscribe once: when the worker drains the queue it dispatches the agent.batch.completed event to every tenant webhook endpoint subscribed to it. Register a webhook endpoint for that event (see Webhooks) and treat the event as the trigger to fetch results with a single GET — instead of hammering the poll endpoint in a loop. The event payload is a notification, not the results. It mirrors what the completed status carries minus the bulky results array:
The webhook emission is best-effort: if your endpoint is unreachable, the results still live in the job and remain fetchable by polling. Treat the webhook as the fast path and the GET as the recovery path.

​
Billing alignment

Every item in a batch is invoked through the same path as a live /chat turn — the same metered agent invocation. Token usage is metered exactly like live chat, the conversation is persisted like a live turn, and the per-item tokens_used and cost_cents in the result set are the same accounting you’d see on a synchronous call. No separate batch pricing exists, and retry-at-the-queue-level is deliberately disabled so a transient failure never double-charges you: retries surface as per-item failures, not silent resubmission. That makes budget guardrails apply end-to-end — a batch runs against your cost controls exactly as equivalent live turns would. See Cost controls to set the spend cap a runaway batch cannot exceed.

​
Result retention

Batch state lives in the queue’s completed job, keyed by batch_id. Completed batches are retained for roughly a week (the newest 500 completed batches are kept); failed batches longer. Poll within that window — an aged-out batch returns 404, and there is no recovery endpoint, so collect results when agent.batch.completed fires rather than days later.

​
Limits and responsibilities

  • 500 items per batch — split larger workloads across multiple submissions. Submit is rate-limited tighter than a normal write (10 requests/minute per token) because one call fans out to many runtime hops.
  • Per-item ceilings — input max 8000 characters, history max 20 turns.
  • 45-second per-item timeout at the worker — an item that exceeds it is recorded as failed; choose live /chat for inputs that need genuinely long-running tool chains.
  • Role gates — submit requires owner, admin, or developer; reading another batch’s results carries the same sensitivity class as submitting, so it is guarded to the same roles.
  • Idempotency — the route fills in custom_id where omitted, and the queue dedupes on batch_id, but submit twice and you run twice. Pass your own Idempotency-Key as on any POST to make retries safe.

​
Next steps