> ## Documentation Index > Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt > Use this file to discover all available pages before exploring further. # Voice gateway: the realtime AI-voice media edge > The voice-gateway service behind every AI voice turn — the STT/TTS provider plug model, boot-time failover chains, per-language provider routing, session pre-warm, barge-in semantics, and the outbound-call origination bridge. # Voice gateway: the realtime AI-voice media edge Every spoken turn of an AI voice agent on Devotel Orbit crosses one realtime edge service: the **voice gateway**. It sits between the carrier network and the agent application, and it is the only place where speech becomes text and text becomes speech. If you build voice agents against Orbit, the gateway is the surface whose provider model, failover contract, and event stream you are actually integrating with — the carrier underneath and the agent above are both replaceable, the contract in the middle is not. ## 1. Where the gateway sits Three planes meet on a voice-agent call: * **The carrier plane** terminates PSTN/SIP media — the Jambonz session border controller with RTPengine bridging carrier RTP (see [Media planes](/concepts/media-planes)). It knows nothing about agents or vendors. * **The voice gateway** is the realtime media edge. It takes the call's audio stream off the carrier plane, runs speech-to-text (STT) on the caller and text-to-speech (TTS) back at them, and exposes the session to the agent application over a typed WebSocket protocol. Provider selection, failover, pre-warm, and barge-in all live here. * **The application plane** is where AI agents run: they consume committed transcripts and tool results and emit replies the gateway speaks back onto the same leg (see [AI agent architecture](/concepts/ai-agent-architecture)). Agents never terminate carrier media themselves. One rule binds all three at the origination boundary: outbound (MT) voice terminates only via the Devotel wholesale softswitch, whether a human, a campaign, or an agent initiated the call. The gateway selects between two SIP trunk identities on that same softswitch — AI-agent and text-to-speech dispatches ride the media trunk, plain non-AI calls prefer the dedicated direct trunk — never an outside carrier. ## 2. The provider-plug model The gateway talks to four cloud STT vendors and three cloud TTS vendors without the session ever seeing a vendor-specific shape. Every vendor implements one of two small interfaces: * **STT** — a provider exposes `transcribe(audio)` for one-shot work and `createStream(options)` for a live session that emits transcript events. Options walk the stream for the call: `language` (or the `multilingual` auto-detect mode when a caller is expected to switch languages mid-call), `model`, `utteranceEndMs` as the turn-boundary latency knob, and per-agent `keyterms` — brand names, SKUs, and vertical jargon the base model would mishear, rendered into each vendor's native vocabulary-bias dial. * **TTS** — a provider exposes `synthesize(text, voiceId)` for one-shot speech and `streamSynthesize(text, voiceId)` for chunked playback, with options for language, speech rate, and a tenant pronunciation dictionary each provider renders into its own markup. The STT vendors are **Deepgram** (Nova-3 streaming, the primary), **OpenAI Whisper**, **Azure Speech**, and **Google Cloud Speech** (batch fallbacks). The TTS vendors are **Cartesia** (Sonic, the primary), **Deepgram** (Aura), and **ElevenLabs** (Flash). A session picks its pair at connect time. The gateway has exactly one active STT provider object and one active TTS provider object at any moment — each is either a single vendor (when keys for only one are configured) or a **failover chain** that behaves like one vendor to everything downstream. Session options travel through the chain unchanged, so a session that crosses vendors mid-call keeps its model, language handling, vocabulary bias, and turn-boundary tuning rather than silently dropping to vendor defaults. The chain itself is built **once at server boot**: every vendor whose API key is set gets a slot; a vendor without credentials is omitted, so a partially configured deployment still gets whatever resilience is available. There is no per-call provider knob — you choose providers by provisioning keys, you pick per-call behaviour with session options. ## 3. The failover contract Both directions of speech use the same failover discipline. The construction order is fixed at boot — Deepgram → Whisper → Azure → Google for STT, Cartesia → Deepgram → ElevenLabs for TTS — ranked by streaming latency and recognition quality. What integrates differently is *when* the chain is allowed to move: * **Pre-stream failure fails over.** If a vendor's stream fails before it has delivered a single transcript (STT) or a single audio chunk (TTS), the chain advances to the next vendor. For STT, the audio the caller had already spoken — buffered up to a 4 MB preroll cap — is replayed into the fallback, so a vendor dying mid-boot does not eat the opening words of the utterance. For TTS, the text restarts from the beginning; nothing was audible yet, so nothing doubles. * **Mid-stream failure does NOT fail over.** Once the vendor has delivered at least one transcript (STT) or chunk (TTS), a failure surfaces as an error on the stream and stops there. Re-opening the session on a fallback vendor would replay or restart speech that was already past the point of delivery: the conversation pipeline would see duplicate or out-of-order transcripts and mis-route the turn, and the caller would hear the reply stutter ("Hello, my n— Hello, my name is Alice"). Partial output with a clean error beats duplicated output. * **One-shot calls fail over through the whole chain.** The batch `transcribe` / `synthesize` paths try every configured vendor in order and rethrow the last error only if all fail. ### Per-language re-partitioning The boot order is optimal for English and the languages Deepgram's Nova-3 multilingual mode covers — but for any other language it pays a near-guaranteed miss against Deepgram before reaching a vendor that claims the language. So on every call, the chain re-partitions itself for the request's `language` tag: * Vendors that **declare support** for the language move to the front, keeping their original relative order. * Vendors that do not move to the back, also keeping their relative order — **they stay in the chain** as the exhaustive fallback net. An incomplete coverage table degrades to "try it anyway eventually", never to "silently drop a vendor". * An unset language, or the `multi` auto-detect mode, leaves the boot order byte-for-byte unchanged, so the default English path is untouched. A vendor missing from the coverage table is treated as broad coverage (fail open — it still gets tried). The result of the partition is a new ordering each call; the boot order itself never mutates. ## 4. Lifecycle semantics ### Session pre-warm The slowest contributor to first-turn latency is the cold LLM. When a session opens, the gateway fires one fire-and-forget pre-warm per session so the first real caller utterance does not pay the cold-start penalty. Pre-warm success is invisible to the caller; pre-warm failure is logged and means the first turn cold-starts — the session still works. ### The three interrupt classes "Interrupt" is not one behaviour — the gateway distinguishes three classes, and each truncates the agent's turn differently: | Class | Trigger | What happens to the turn | | ----------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Caller barge-in** | The caller speaks over the agent | Playback truncates where the caller actually heard (`audio.flush` with `cause="barge_in"`), and the LLM context truncates to what was delivered. The next turn references only content the caller actually heard — the model never claims it said something the caller missed. | | **Programmatic cancel** | Your application kills the turn (guardrail, retry, plan change) | `audio.flush` with `cause="cancel"`; the agent's output is discarded entirely. No context from the aborted turn is preserved. | | **Human takeover** | A supervisor cuts in | `audio.flush` with `cause="takeover"`; generation pauses while the human holds the call. If the takeover was requested as resumable, release can hand the call back with a context note describing what was said while the agent was paused. | The wire-level events (`agent.barge_in`, `agent.cancel`, `supervisor.takeover`, `supervisor.release`, `audio.flush`) are defined in the [Orbit Voice Protocol](/api-reference/orbit-voice); this contract is the reason it splits interrupts into separate typed events instead of overloading one. ### The outbound-call origination bridge The gateway owns the dial-plan boundary for outbound voice at `POST /api/v1/outbound-call`. In order, and before anything becomes billable media: 1. **Validation and safety gates** — emergency short codes are rejected up front; destination coverage is advisory (the softswitch delivers a real SIP error if a destination is unreachable; per-tenant destination restrictions belong in tenant settings, not a platform-wide gate). 2. **Compliance gates you configure** — TCPA quiet-hours, Do-Not-Call lists, and cross-channel STOP honours are enforced against your tenant configuration; a call that cannot be compliance-checked fails closed rather than dialling unchecked. AI-synthesized and cloned-voice content is additionally gated on the written-consent state you maintain for that voice content. 3. **Origination** — the gateway picks the SIP trunk (media trunk for agent/TTS dispatches, the dedicated direct trunk for plain non-AI calls) and dispatchers out through Jambonz to the Devotel wholesale softswitch. STIR/SHAKEN attestation and, when you have enabled branded calling, Rich Call Data presentation are signalled to the softswitch, which signs at the level your caller ID qualifies for. A signalling miss degrades the attestation level; it never blocks the call. 4. **Recording and consent** — `record-from-answer` starts audio recording once the callee answers, scoped to your tenant's storage region. If your organization requires a recording-consent announcement, the gateway speaks the banner BEFORE recording starts, and recording is skipped entirely (fail-closed) when the banner cannot play — a silently recorded call in a two-party-consent context is the worst outcome, so the recording is sacrificed instead. 5. **Special dispatches** — `agent_id` mints the media room the agent pipeline joins; a one-way text payload (`metadata.tts_text`) schedules a text-to-speech blast that speaks once the callee picks up and then hangs the call up; `dynamic_variables` templates the greeting and blast text per lead. Answering-machine detection (`amd: true`) attaches detection so machine answers end the attempt rather than burn agent time. The response (`201`) returns call identifiers plus `room_name` (the media room), `sip_call_id`, recording ids when started, and flags such as `amd_enabled` / `tts_blast` you can record without re-parsing your own request. ## 5. What integrators branch on * **`transcript.partial` vs `transcript.final`.** Streaming vendors emit interim hypotheses and committed finals; batch fallbacks emit only finals. Branch rendering and conversational memory on finals; use partials for live typing indicators at most. Never treat an interim transcript as the caller's utterance. * **`audio.flush` causes.** The flush event's `cause` field (`barge_in` / `cancel` / `takeover` / `session_end`) tells you which interrupt class happened — drive your UI and analytics off it instead of inferring from audio stops. * **Failover observability.** The failover happens in the gateway's provider chain; your visible signal is on the event stream — an STT/TTS upstream error event, then a recovered session (pre-stream) or an ended one (mid-stream). If a session opens in a language Deepgram does not cover well, expect the chain to start one tier down by design (see per-language re-partitioning), not by failure. * **Session resume.** If the WebSocket drops within the resume window, reconnect with `session.resume`; the gateway replays what you missed and the agent picks up in place. ## Worked example: one inbound turn with a Deepgram → Whisper boot failover A caller dials a DID routed to your agent. The session has already been created by the inbound bridge; your client connects and the first Deepgram stream fails before any transcript — the chain advances to Whisper and replays the buffered audio so the caller's opening words survive: ``` [server→client] session.start seq=0 agent.language="en-US" tools=[…] [client→server] (binary audio frames @ 20ms, direction=0, frame_seq 0..N) — Deepgram stream errors before its first transcript; gateway replays the buffered preroll into Whisper — [server→client] audio.ingress seq=1 [server→client] transcript.partial seq=2 "What time" [server→client] transcript.partial seq=3 "What time do you close" [server→client] transcript.final seq=4 "What time do you close on Sundays?" [server→client] agent.thinking seq=5 delta="We" [server→client] agent.thinking seq=6 delta=" close" [server→client] agent.thinking seq=7 delta=" at" [server→client] agent.thinking seq=8 delta=" 6pm" [server→client] agent.output seq=9 "We close at 6pm on Sundays." final=true [server→client] audio.egress seq=10 utterance=utt_a bytes_sent=12800 final=false [server→client] (binary audio frames, direction=1) [server→client] audio.egress seq=11 utterance=utt_a bytes_sent=32000 final=true [server→client] agent.latency.breakdown seq=12 [server→client] session.end seq=13 reason=caller_hangup ``` From your client's perspective nothing about the failover is visible on the wire except a later-than-usual first partial. Inside the gateway, the Deepgram leg failed pre-stream, the preroll buffer replayed into Whisper, the preroll sealed on Whisper's first transcript, and the rest of the session runs on the fallback with the same session options. Had Deepgram failed *after* the first partial, you would instead have seen an `error` event with `OVP_STT_UPSTREAM_FAILED` and no failover — that is the duplicate-transcript trade-off from section 3 showing up on the wire. The wire format itself (event envelopes, binary frame header, resume semantics) is specified in the [Orbit Voice Protocol](/api-reference/orbit-voice). ## Cross-references * [Media planes](/concepts/media-planes) — the two-plane split (SFU vs carrier) the gateway sits across. * [AI agent architecture](/concepts/ai-agent-architecture) — the application plane the gateway serves. * [Agent run lifecycle](/concepts/agent-run-lifecycle) — the state machine every turn the gateway bridges moves through. * [Orbit Voice Protocol](/api-reference/orbit-voice) — the wire-level event and frame contract. * [Voice call lifecycle](/concepts/voice-call-lifecycle) — the call-level lifecycle surrounding the session.