> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Tune the semantic response cache: enablement, threshold, and TTL from your traffic

> Turn the semantic response cache on in Settings → Agents or over the settings API, pick a similarity threshold and a TTL that fit your opening traffic, and verify on the usage page that hits are saving model calls.

# Tune the semantic response cache

The semantic response cache lets an agent answer a near-duplicate opening question from a recently stored answer instead of running a fresh model call. This guide walks the three decisions you make as an operator: whether to enable it, how tight to set the similarity threshold, and how long a stored answer may live. Then it shows how to verify a hit actually saved a generation, and which traffic shapes should leave the cache off.

The model behind the cache — eligibility rules, ring mechanics, failure posture — is covered in [Semantic response cache](https://docs.orbit.devotel.io/concepts/semantic-response-cache-model.md). This guide stays on the tuning decisions.

## Prerequisites

* An Orbit account with the owner role to change the setting. Developer and above can read it.
* An agent whose conversations open with questions from a small, stable set: password resets, opening hours, plan prices, order status.
* Access to Billing → Usage → AI ([AI usage breakdown](https://docs.orbit.devotel.io/guides/billing-ai-usage-split.md)) for the cost read after you enable.

## 1. The cache in one screen

When a conversation opens, the runtime embeds the first message once and compares that vector against a small ring of recent question-and-answer pairs kept for that agent. If the closest entry clears your similarity threshold, the stored answer is served and the model call, the tool loop, and any knowledge retrieval behind it are skipped. If nothing clears the threshold, the normal agent path runs, and the fresh pair is stored at the head of the ring for the next near-duplicate question.

Three rules define the cache's shape:

* **First turn only.** A conversation with any message history runs the normal path; the cache never substitutes for a follow-up.
* **One ring per agent.** Entries are keyed by your organization and the agent, capped at the 50 most recent pairs, and the whole ring expires when the TTL lapses.
* **A cosine similarity threshold you own.** A hit requires the incoming question's embedding to score at or above your threshold against a stored question. The default, 0.95, means near-paraphrase.

Two adjacent caches already exist and are easy to confuse with this one:

| Cache | Matches | Saves |
| - | - | - |
| Embedding cache | Identical text | The embedding call only; a reworded question is embedded again |
| Exact-match answer cache | A character-for-character identical question | The generation, but only on a verbatim repeat |
| Semantic response cache (this guide) | The same meaning in different words | The generation, the tool loop, and retrieval |

Real FAQ traffic lives between exact and semantic: customers ask the same thing in different words ("How do I reset my password?", "I forgot my password"). If your openings repeat verbatim, the exact-match cache already covers them; the semantic cache earns its keep on paraphrase.

## 2. Enable it

### In the dashboard (owner)

Open **Settings → Agents** and switch the response cache on. The threshold and lifetime fields are optional: leave them blank to keep the platform defaults (0.95 and 3,600 seconds). Changes take effect on the next agent turn.

### Over the API

```bash theme={null}
curl -X PUT https://orbit.devotel.io/api/v1/settings/agents/semantic-cache \
  -H "Authorization: Bearer <YOUR_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "threshold": 0.95,
    "ttl_seconds": 3600
  }'
```

`enabled` is the only required field. Omit `threshold` or `ttl_seconds`, or send them as `null`, to clear an override and fall back to the defaults. The write requires the owner role; a developer-role token can still read the stored configuration:

```bash theme={null}
curl https://orbit.devotel.io/api/v1/settings/agents/semantic-cache \
  -H "Authorization: Bearer <YOUR_API_TOKEN>"
```

The read returns what you stored; a field you left to the default comes back as `null`, and the runtime applies the default for it.

### Validation and clamping behaviour

* The API rejects out-of-band values with a 422: `threshold` must lie between 0.8 and 0.9999, and `ttl_seconds` must be an integer between 60 and 86,400.
* A value that reaches the settings store through another write path is clamped at read time: a threshold below 0.8 is raised to the floor, one above 0.9999 lowered to the ceiling, and a TTL outside 60 to 86,400 seconds is clamped into that band. A missing or non-numeric value falls back to the default.
* Disabling is clean. When `enabled` is not exactly `true`, the runtime skips the cache entirely: no embedding lookup, no behaviour change.

## 3. Choose a threshold and a TTL

Start at the default 0.95 and move only when your traffic gives you a reason.

**When to loosen (toward 0.90).** FAQ-heavy support bots where one intent arrives in many surface forms: "reset my password", "forgot my password", "password reset steps". At 0.90, rewordings that carry the same intent clear the bar and the hit rate climbs. When you loosen, run the spot-checks in section 4: each step down trades hit rate for the risk of serving a stored answer to a question that only almost matches. The 0.8 floor is deliberate; below it, distinct questions start to collide, and the API will not accept a lower value.

**When to tighten (toward 0.97 to 0.99).** Agents whose answers carry regulated advice, prices, entitlements, or eligibility rules, anywhere a near-miss answer is worse than a freshly generated one. A tighter band means fewer hits and more billed generations; that is the cache working as designed, trading savings for certainty.

**TTL by content freshness.** The TTL is how long a stored answer may be served. It re-arms on every store, so an actively hit ring keeps its entries alive; the age that matters is the gap between stores.

| Answer content | Suggested TTL |
| - | - |
| Prices, stock, or policy that changes with releases | Hours: the 3,600 second default, or shorter (minimum 60) around known change windows |
| Stable how-to and policy answers | Up to the 86,400 second (one day) ceiling |

A cache hit serves exactly what was stored: no personalization and no fresh data. If the underlying answer changed, the ring serves the old wording until the TTL lapses or a miss overwrites the entry. Size the TTL to the age at which an answer becomes wrong, not the age at which it becomes unpopular.

## 4. Verify it works

1. **Read the configuration back.** `GET /api/v1/settings/agents/semantic-cache` returns the stored values; `null` means the default applies. Confirm `enabled: true` before reading anything into the numbers below.
2. **Watch the agent's cost row.** On Billing → Usage → AI ([AI usage breakdown](https://docs.orbit.devotel.io/guides/billing-ai-usage-split.md)), a cache-served opening is billed zero tokens for the generation while reporting a consistent estimated usage count, so the row never reads as an unexplained zero. With conversation volume steady, a working cache shows up as the agent's cost and cost per conversation trending down. Note that the page's **Cache hit** column measures the provider's prompt cache, a separate automatic lever, not this cache; read cost per conversation for the semantic cache's effect.
3. **Spot-check served answers.** Sample opening turns and confirm a served answer matches what the agent says elsewhere for the same question, and that it is self-contained: no reference to contact data, no tool output, no escalation that should have fired. The store side already refuses answers that called tools or escalated, and the lookup side refuses openings that match an escalation trigger, so a served answer should always be a clean standalone reply.
4. **Confirm a miss stores a fresh pair.** After a miss on a clean first turn, a near-paraphrase of that question should hit inside the TTL. If paraphrases keep missing, either the ring's 50 entries are dominated by other questions or the threshold is too tight for your traffic's phrasing spread.

For what a saved generation is worth against your volume, run the cost math in [Agent ROI attribution](https://docs.orbit.devotel.io/guides/agent-roi-attribution.md).

## 5. When not to enable it

The platform already excludes the dangerous cases: agents with per-contact memory, agents with cross-channel continuity, openings with attachments, and escalation-bound openings are never eligible, and answers that called tools or escalated are never stored. What it cannot decide for you is whether your traffic has anything to hit:

* **Context-dependent openings.** If first messages are usually continuations ("following up on my ticket", "as I said yesterday"), first-turn-only eligibility leaves almost nothing eligible, and what is eligible is too contextual to serve from a stored answer.
* **Per-customer data in the first answer.** If a correct opening reply depends on who is asking (balance, tier, open tickets), a cached answer produced for another contact is wrong by construction. Per-contact memory is the right tool, and the two are mutually exclusive by design.
* **Tool-loop-dependent responses.** If most openings fire a tool call (a lookup, a booking, a calculation), those answers are never stored, and the cache sits idle while adding one embedding lookup per opening.

In all three the safe posture is to leave `enabled` off. The feature is opt-in and fail-closed, and turning it off removes the lookup entirely.

## Related reading

* [Semantic response cache](https://docs.orbit.devotel.io/concepts/semantic-response-cache-model.md) — the model: eligibility, ring mechanics, failure posture.
* [AI usage breakdown](https://docs.orbit.devotel.io/guides/billing-ai-usage-split.md) — the usage page used in section 4.
* [Agent ROI attribution](https://docs.orbit.devotel.io/guides/agent-roi-attribution.md) — the cost math for a saved generation.
* [Cost controls](https://docs.orbit.devotel.io/agents/cost-controls.md) — the ceilings that bound what a cache-missed run may spend.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.