Skip to main content

​
The knowledge-base ingestion pipeline

Uploading a document to a knowledge base returns immediately with the document in a processing state. Between that response and the moment the document can ground an agent’s answer, an asynchronous machine does the real work: extract text, chunk, embed, and store vectors under a bounded, fair, retry-safe schedule. The Build and Maintain an AI Knowledge Base guide covers what you do with the API — this page explains what the platform does with your document after the API hands it off, and why it is built the way it is. The distinction matters operationally: when a document appears stuck, the fix depends on which stage of the machine is holding it, and every stage is designed to be safe to retry.

​
1. Upload → processing → the scheduler tick

Every document you add — a file upload, a JSON text post, or a URL source — lands in the knowledge base with status processing. The upload endpoint does no embedding inline for one reason: extraction and embedding take seconds, and an interactive request should never carry a seconds-long, rate-limited pipeline in its path. A platform scheduler wakes on a short tick (under a minute), scans the processing queue in every workspace, oldest submission first, and ingests what it finds. The end-to-end outcome you should expect: upload → ready in roughly a minute or two in a healthy system. For URL documents, the row also carries the crawl configuration (depth, connector binding, per-document audience access lists) you set at upload, and the ingestion pass honors it when the tick picks the document up.

​
2. Idempotent re-ingest: retries never duplicate

The ingestion service is idempotent by design: before a document is re-indexed, its prior chunks and vectors are cleared, and only then is the new index written. Retrying a document therefore rebuilds the same final state rather than stacking a second copy on top — you never retrieve duplicated chunks because the scheduler had to take a second pass. That property carries the whole error-handling model. A worker shut down mid-pipeline, a retry you trigger from POST /knowledge-bases/:id/documents/:docId/retry, or a fresh ingest following heartbeat loss — all of them simply re-run the same clearing + re-indexing pass. Interruption safety is structural, not incidental: whatever stage a document stops at, the next pass reaches the same terminal state from a clean slate.

​
3. Scheduling: how the pipeline shares capacity fairly

Three scheduling decisions shape what you observe during large imports:
  • Tenants run in parallel; one tenant’s documents run serially. The embedding provider rate-limits per API project, not per tenant — so serializing one workspace’s documents is deliberate back-pressure, while parallelizing across workspaces preserves throughput for the fleet.
  • Every tick is bounded. A fixed number of documents is ingested per tick overall, with a smaller per-tenant slice, so a bulk import from one workspace cannot starve everyone else’s uploads. The rest of a large batch simply waits for the next tick — upload order is preserved oldest-first, so nothing re-queues out of turn.
  • Shutdown is graceful mid-tick. On a rolling deploy the scheduler stops picking up new work at the next tenant boundary rather than dropping in-flight work mid-pipeline; anything left unstarted lands in the next tick’s queue without manual intervention.
These limits are why a 500-document import trickles in as ready steadily rather than as one single batch flip — the fairness ceiling shows up to you as a queue draining at a predictable rate.

​
4. Terminal states: ready and failed

Each document ends in one of two terminal states:
  • ready — the chunks are embedded, indexed, and visible to search_knowledge retrieval on the next agent turn. A successful ingest also records the ingestion as a usage event, so ingestion activity you perform actually shows up in workspace analytics.
  • failed — the pipeline could not produce a complete index. The failure is written onto the document with a short reason, and the scheduler verifies the terminal state after every unsuccessful attempt so a permanently-failing document cannot silently loop in processing forever. Causes fall into a few practical buckets: an unreadable upload (truncated file, zero-byte file, unsupported content), an OCR pass that extracted no text, an unreachable or redirecting URL, or a document larger than the 10 MB content limit.
Because re-ingest is idempotent, failed is not an end state you are stuck with: the retry endpoint re-runs the ingestion against the same source bytes and replaces the outcome cleanly. A retry against genuinely broken source bytes will fail identically — in that case delete the document and re-upload the corrected file.

​
5. Freshness after ingest: recrawl, refresh, and versioning

Ingestion is the write-once path; the platform layers three more machines on top to handle content that changes after it lands:
  • Recrawl (URL sources). URL documents are otherwise scrape-once: nothing re-fetches them inherently. The recrawl scheduler re-fetches URL sources on a platform-configurable interval (default daily, with an hourly floor). A recrawl compares the fetched content against what’s indexed and short-circuits when nothing changed — an unchanged recrawl embeds nothing. Each recrawl (successful or failed) advances the document’s freshness clock, so a permanently broken source cannot spin in a tight retry loop; on your side, the per-document staleness badge in the dashboard reads from that same clock.
  • Scheduled refresh. A base created with refresh_schedule (hourly, daily, weekly) opts its external-source documents (URL, RSS, Notion) into recurring re-fetch with exponential backoff on failure. Refresh is the base-level counterpart to the per-document recrawl — an individual URL document is kept current by recrawl on the platform cadence; setting a base’s refresh schedule concentrates the same behavior as a workspace-scoped policy you control.
  • KB versioning. Re-uploading a document with the same name replaces the prior version — its chunks and vectors are cleared under the same idempotent re-ingest contract. Without a safety net, a bad replacement would silently degrade answers with no way back. Versioning prevents that: every re-upload captures an immutable snapshot of the superseded version (the ten most recent are retained), so you can diff the new content against the old and, where the snapshot is under the retention ceiling, roll back to a known-good version through GET /knowledge-bases/:id/documents/:docId/versions and POST .../rollback. Version history composes with recrawl and refresh rather than gates it: the history records what replaced what, whichever path replaced it.
Connectors (Notion, Confluence, Google Drive, SharePoint, Zendesk) also ride this pipeline: a connector sync resolves content through your authorized integration, and the documents it lands are ingested, re-ingested, and versioned by the same machines above. The full authoring surface, drift reports, and cadence options are in the lifecycle guide.

​
6. Where ingestion ends and retrieval begins

The ingestion pipeline’s job stops at indexed; retrieval is a separate machine that runs at answer time. Attached to an agent, a ready knowledge base becomes the corpus for the search_knowledge tool: each conversation turn embeds the customer’s question and matches it against the stored chunks, then grounds the model’s reply in the top-scoring matches. Everything above — the idempotent re-ingest, the freshness schedulers, the version history — exists so that what retrieval finds at answer time is current, complete, and never duplicated. The retrieval side itself, including how retrieved chunks enter the prompt, is covered in AI agent architecture §2.

​
Cross-references