> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Async batch inference for AI agents

> Run hundreds of agent invocations asynchronously — classification, summarization, labeling, CDP enrichment — with POST /agents/:id/batch, poll GET /agents/:id/batch/:batchId, or subscribe to the agent.batch.completed webhook.

# Async batch inference for AI agents

The batch inference API lets you submit up to 500 agent invocations in one call and collect the results asynchronously. It exists for workloads where a synchronous `/chat` turn is the wrong shape: anything where you have a pile of inputs, you want the agent's judgment on each one, and you don't need an answer inside the request/response.

Typical workloads:

* **Mass classification** — label 400 inbound messages by intent or sentiment for a routing model.
* **Summarization** — condense yesterday's resolved conversations into nightly digests.
* **Data labeling** — generate training pairs for an internal model.
* **CDP enrichment** — extract a preference or attribute from each contact's latest thread and write it back to the contact record.

Use live `/chat` when a person or a UI is waiting on the reply. Use the batch when throughput matters and latency doesn't.

## How a batch works

A batch is submitted with one POST, processed item by item by a background worker, and retained so you can collect results when it finishes:

1. **Submit** — `POST /agents/:id/batch` validates and queues the batch, and answers `202 Accepted` with a `batch_id` immediately.
2. **Track** — `GET /agents/:id/batch/:batchId` returns the live status (`queued` → `processing` → `completed`/`failed`) plus progress counts while the worker drains the queue.
3. **Collect** — when the batch completes, either poll the same GET or subscribe to the `agent.batch.completed` tenant webhook, then fetch the full per-item result set.

Submit and read require owner, admin, or developer roles and the `agents:write` / `agents:read` scopes respectively.

## Submit a batch

`POST /api/v1/agents/:id/batch` accepts up to 500 items. Each item carries the message to send the agent, an optional prior-turn history (up to 20 turns) so an item can continue a thread, and an optional `custom_id` (up to 120 chars) that is echoed back on the result so you can join the outcome to your own records. Any item without a `custom_id` gets a generated one. `custom_id` must be unique within the batch — a duplicate is rejected with `422 BATCH_DUPLICATE_CUSTOM_ID` so the result set stays unambiguous.

```bash theme={null}
curl -X POST 'https://api.orbit.devotel.io/api/v1/agents/agt_01J8Z9K3P4Q5R6S7T8U9V0W1X2/batch' \
  -H 'Authorization: Bearer '$(get_jwt) \
  -H 'Content-Type: application/json' \
  -d '{
    "items": [
      {
        "custom_id": "msg_8841/intent",
        "input": "Classify the intent of this message in one word: \"Where is my invoice for May?\""
      },
      {
        "custom_id": "msg_8842/intent",
        "input": "Classify the intent of this message in one word: \"I want to cancel my subscription.\""
      }
    ]
  }'
```

The response is `202 Accepted` — the batch is queued, not started:

```json theme={null}
{
  "data": {
    "batch_id": "agentBatch_01J8Z9K3P4Q5R6S7T8U9V0W1X2",
    "agent_id": "agt_01J8Z9K3P4Q5R6S7T8U9V0W1X2",
    "status": "queued",
    "total_items": 2,
    "agent_version_id": null
  },
  "meta": { "request_id": "req_01J8", "timestamp": "2026-09-18T09:14:02.510Z" }
}
```

Error surface at submit:

* `404 NOT_FOUND` — the agent does not exist in your tenant; the batch is not queued.
* `422 BATCH_DUPLICATE_CUSTOM_ID` — an item repeats a `custom_id`; resubmit with unique ids.
* `422 VALIDATION_ERROR` — the body failed validation (over 500 items, an `input` over 8000 characters, more than 20 history turns).
* `503 BATCH_QUEUE_UNAVAILABLE` — the batch queue is temporarily unavailable; nothing was accepted, so retry shortly.

To pin every item to a specific published version of the agent, pass `agent_version_id` at the top level. Omit it and the batch runs against the agent's current candidate version.

## Poll for status and results

`GET /api/v1/agents/:id/batch/:batchId` reports the batch's current state. While the worker is draining the queue you get live progress; when it finishes the same response carries the aggregate summary and the full per-item result set.

```bash theme={null}
curl -s 'https://api.orbit.devotel.io/api/v1/agents/agt_01J8Z9K3P4Q5R6S7T8U9V0W1X2/batch/agentBatch_01J8Z9K3P4Q5R6S7T8U9V0W1X2' \
  -H 'Authorization: Bearer '$(get_jwt)
```

In flight:

```json theme={null}
{
  "data": {
    "batch_id": "agentBatch_01J8Z9K3P4Q5R6S7T8U9V0W1X2",
    "agent_id": "agt_01J8Z9K3P4Q5R6S7T8U9V0W1X2",
    "status": "processing",
    "total_items": 400,
    "progress": { "completed": 128, "succeeded": 126, "failed": 2 },
    "summary": null,
    "results": null,
    "error": null
  }
}
```

Completed — `status` flips to `completed`, `summary` rolls up the roll-call, and `results` holds one entry per submitted item in submission order:

```json theme={null}
{
  "data": {
    "batch_id": "agentBatch_01J8Z9K3P4Q5R6S7T8U9V0W1X2",
    "agent_id": "agt_01J8Z9K3P4Q5R6S7T8U9V0W1X2",
    "status": "completed",
    "total_items": 400,
    "progress": { "completed": 400, "succeeded": 396, "failed": 4 },
    "summary": {
      "total": 400,
      "succeeded": 396,
      "failed": 4,
      "total_tokens": 312480,
      "total_cost_cents": 18.7742
    },
    "results": [
      {
        "custom_id": "msg_8841/intent",
        "status": "succeeded",
        "response": "billing_inquiry",
        "tokens_used": 781,
        "cost_cents": 0.0469,
        "model": "claude-sonnet-4-5"
      },
      {
        "custom_id": "msg_8842/intent",
        "status": "failed",
        "tokens_used": 0,
        "cost_cents": 0,
        "error": "Agent-runtime chat returned 429: rate limit"
      }
    ],
    "error": null
  }
}
```

A per-item failure never sinks the batch: failed items are recorded with an `error` message and zero usage, and the batch still completes. Only an infrastructure-level failure flips the whole batch to `failed`, in which case `error` carries the job-level reason.

Tenant isolation: a batch id resolves only for the tenant and agent that submitted it. A valid id polled by another tenant, or under another agent, returns `404` — never the other tenant's results.

## The completion webhook

Polling works, but the better pattern is to subscribe once: when the worker drains the queue it dispatches the `agent.batch.completed` event to every tenant webhook endpoint subscribed to it. Register a webhook endpoint for that event (see [Webhooks](/webhooks/events)) and treat the event as the trigger to fetch results with a single GET — instead of hammering the poll endpoint in a loop.

The event payload is a notification, not the results. It mirrors what the completed status carries minus the bulky `results` array:

```json theme={null}
{
  "event": "agent.batch.completed",
  "tenant": "tn_01J8Z9K3P4Q5R6S7T8U9V0W1X2",
  "data": {
    "batch_id": "agentBatch_01J8Z9K3P4Q5R6S7T8U9V0W1X2",
    "agent_id": "agt_01J8Z9K3P4Q5R6S7T8U9V0W1X2",
    "status": "completed",
    "summary": {
      "total": 400,
      "succeeded": 396,
      "failed": 4,
      "total_tokens": 312480,
      "total_cost_cents": 18.7742
    },
    "completed_at": "2026-09-18T09:19:47.842Z"
  }
}
```

The webhook emission is best-effort: if your endpoint is unreachable, the results still live in the job and remain fetchable by polling. Treat the webhook as the fast path and the GET as the recovery path.

## Billing alignment

Every item in a batch is invoked through the same path as a live `/chat` turn — the same metered agent invocation. Token usage is metered exactly like live chat, the conversation is persisted like a live turn, and the per-item `tokens_used` and `cost_cents` in the result set are the same accounting you'd see on a synchronous call. No separate batch pricing exists, and retry-at-the-queue-level is deliberately disabled so a transient failure never double-charges you: retries surface as per-item failures, not silent resubmission.

That makes budget guardrails apply end-to-end — a batch runs against your cost controls exactly as equivalent live turns would. See [Cost controls](/agents/cost-controls) to set the spend cap a runaway batch cannot exceed.

## Result retention

Batch state lives in the queue's completed job, keyed by `batch_id`. Completed batches are retained for roughly a week (the newest 500 completed batches are kept); failed batches longer. Poll within that window — an aged-out batch returns `404`, and there is no recovery endpoint, so collect results when `agent.batch.completed` fires rather than days later.

## Limits and responsibilities

* **500 items per batch** — split larger workloads across multiple submissions. Submit is rate-limited tighter than a normal write (10 requests/minute per token) because one call fans out to many runtime hops.
* **Per-item ceilings** — `input` max 8000 characters, `history` max 20 turns.
* **45-second per-item timeout** at the worker — an item that exceeds it is recorded as failed; choose live `/chat` for inputs that need genuinely long-running tool chains.
* **Role gates** — submit requires owner, admin, or developer; reading another batch's results carries the same sensitivity class as submitting, so it is guarded to the same roles.
* **Idempotency** — the route fills in `custom_id` where omitted, and the queue dedupes on `batch_id`, but submit twice and you run twice. Pass your own `Idempotency-Key` as on any POST to make retries safe.

## Next steps

* [Cost controls](/agents/cost-controls) — cap the spend a batch can consume.
* [Webhooks](/webhooks/events) — register `agent.batch.completed` on your tenant.
* [Agents API reference](/api-reference/endpoints/agents) — full parameter and schema detail for both endpoints.