Skip to main content

​
Voice gateway: the realtime AI-voice media edge

Every spoken turn of an AI voice agent on Devotel Orbit crosses one realtime edge service: the voice gateway. It sits between the carrier network and the agent application, and it is the only place where speech becomes text and text becomes speech. If you build voice agents against Orbit, the gateway is the surface whose provider model, failover contract, and event stream you are actually integrating with — the carrier underneath and the agent above are both replaceable, the contract in the middle is not.

​
1. Where the gateway sits

Three planes meet on a voice-agent call:
  • The carrier plane terminates PSTN/SIP media — the Jambonz session border controller with RTPengine bridging carrier RTP (see Media planes). It knows nothing about agents or vendors.
  • The voice gateway is the realtime media edge. It takes the call’s audio stream off the carrier plane, runs speech-to-text (STT) on the caller and text-to-speech (TTS) back at them, and exposes the session to the agent application over a typed WebSocket protocol. Provider selection, failover, pre-warm, and barge-in all live here.
  • The application plane is where AI agents run: they consume committed transcripts and tool results and emit replies the gateway speaks back onto the same leg (see AI agent architecture). Agents never terminate carrier media themselves.
One rule binds all three at the origination boundary: outbound (MT) voice terminates only via the Devotel wholesale softswitch, whether a human, a campaign, or an agent initiated the call. The gateway selects between two SIP trunk identities on that same softswitch — AI-agent and text-to-speech dispatches ride the media trunk, plain non-AI calls prefer the dedicated direct trunk — never an outside carrier.

​
2. The provider-plug model

The gateway talks to four cloud STT vendors and three cloud TTS vendors without the session ever seeing a vendor-specific shape. Every vendor implements one of two small interfaces:
  • STT — a provider exposes transcribe(audio) for one-shot work and createStream(options) for a live session that emits transcript events. Options walk the stream for the call: language (or the multilingual auto-detect mode when a caller is expected to switch languages mid-call), model, utteranceEndMs as the turn-boundary latency knob, and per-agent keyterms — brand names, SKUs, and vertical jargon the base model would mishear, rendered into each vendor’s native vocabulary-bias dial.
  • TTS — a provider exposes synthesize(text, voiceId) for one-shot speech and streamSynthesize(text, voiceId) for chunked playback, with options for language, speech rate, and a tenant pronunciation dictionary each provider renders into its own markup.
The STT vendors are Deepgram (Nova-3 streaming, the primary), OpenAI Whisper, Azure Speech, and Google Cloud Speech (batch fallbacks). The TTS vendors are Cartesia (Sonic, the primary), Deepgram (Aura), and ElevenLabs (Flash). A session picks its pair at connect time. The gateway has exactly one active STT provider object and one active TTS provider object at any moment — each is either a single vendor (when keys for only one are configured) or a failover chain that behaves like one vendor to everything downstream. Session options travel through the chain unchanged, so a session that crosses vendors mid-call keeps its model, language handling, vocabulary bias, and turn-boundary tuning rather than silently dropping to vendor defaults. The chain itself is built once at server boot: every vendor whose API key is set gets a slot; a vendor without credentials is omitted, so a partially configured deployment still gets whatever resilience is available. There is no per-call provider knob — you choose providers by provisioning keys, you pick per-call behaviour with session options.

​
3. The failover contract

Both directions of speech use the same failover discipline. The construction order is fixed at boot — Deepgram → Whisper → Azure → Google for STT, Cartesia → Deepgram → ElevenLabs for TTS — ranked by streaming latency and recognition quality. What integrates differently is when the chain is allowed to move:
  • Pre-stream failure fails over. If a vendor’s stream fails before it has delivered a single transcript (STT) or a single audio chunk (TTS), the chain advances to the next vendor. For STT, the audio the caller had already spoken — buffered up to a 4 MB preroll cap — is replayed into the fallback, so a vendor dying mid-boot does not eat the opening words of the utterance. For TTS, the text restarts from the beginning; nothing was audible yet, so nothing doubles.
  • Mid-stream failure does NOT fail over. Once the vendor has delivered at least one transcript (STT) or chunk (TTS), a failure surfaces as an error on the stream and stops there. Re-opening the session on a fallback vendor would replay or restart speech that was already past the point of delivery: the conversation pipeline would see duplicate or out-of-order transcripts and mis-route the turn, and the caller would hear the reply stutter (“Hello, my n— Hello, my name is Alice”). Partial output with a clean error beats duplicated output.
  • One-shot calls fail over through the whole chain. The batch transcribe / synthesize paths try every configured vendor in order and rethrow the last error only if all fail.

​
Per-language re-partitioning

The boot order is optimal for English and the languages Deepgram’s Nova-3 multilingual mode covers — but for any other language it pays a near-guaranteed miss against Deepgram before reaching a vendor that claims the language. So on every call, the chain re-partitions itself for the request’s language tag:
  • Vendors that declare support for the language move to the front, keeping their original relative order.
  • Vendors that do not move to the back, also keeping their relative order — they stay in the chain as the exhaustive fallback net. An incomplete coverage table degrades to “try it anyway eventually”, never to “silently drop a vendor”.
  • An unset language, or the multi auto-detect mode, leaves the boot order byte-for-byte unchanged, so the default English path is untouched.
A vendor missing from the coverage table is treated as broad coverage (fail open — it still gets tried). The result of the partition is a new ordering each call; the boot order itself never mutates.

​
4. Lifecycle semantics

​
Session pre-warm

The slowest contributor to first-turn latency is the cold LLM. When a session opens, the gateway fires one fire-and-forget pre-warm per session so the first real caller utterance does not pay the cold-start penalty. Pre-warm success is invisible to the caller; pre-warm failure is logged and means the first turn cold-starts — the session still works.

​
The three interrupt classes

“Interrupt” is not one behaviour — the gateway distinguishes three classes, and each truncates the agent’s turn differently: The wire-level events (agent.barge_in, agent.cancel, supervisor.takeover, supervisor.release, audio.flush) are defined in the Orbit Voice Protocol; this contract is the reason it splits interrupts into separate typed events instead of overloading one.

​
The outbound-call origination bridge

The gateway owns the dial-plan boundary for outbound voice at POST /api/v1/outbound-call. In order, and before anything becomes billable media:
  1. Validation and safety gates — emergency short codes are rejected up front; destination coverage is advisory (the softswitch delivers a real SIP error if a destination is unreachable; per-tenant destination restrictions belong in tenant settings, not a platform-wide gate).
  2. Compliance gates you configure — TCPA quiet-hours, Do-Not-Call lists, and cross-channel STOP honours are enforced against your tenant configuration; a call that cannot be compliance-checked fails closed rather than dialling unchecked. AI-synthesized and cloned-voice content is additionally gated on the written-consent state you maintain for that voice content.
  3. Origination — the gateway picks the SIP trunk (media trunk for agent/TTS dispatches, the dedicated direct trunk for plain non-AI calls) and dispatchers out through Jambonz to the Devotel wholesale softswitch. STIR/SHAKEN attestation and, when you have enabled branded calling, Rich Call Data presentation are signalled to the softswitch, which signs at the level your caller ID qualifies for. A signalling miss degrades the attestation level; it never blocks the call.
  4. Recording and consent — record-from-answer starts audio recording once the callee answers, scoped to your tenant’s storage region. If your organization requires a recording-consent announcement, the gateway speaks the banner BEFORE recording starts, and recording is skipped entirely (fail-closed) when the banner cannot play — a silently recorded call in a two-party-consent context is the worst outcome, so the recording is sacrificed instead.
  5. Special dispatches — agent_id mints the media room the agent pipeline joins; a one-way text payload (metadata.tts_text) schedules a text-to-speech blast that speaks once the callee picks up and then hangs the call up; dynamic_variables templates the greeting and blast text per lead. Answering-machine detection (amd: true) attaches detection so machine answers end the attempt rather than burn agent time.
The response (201) returns call identifiers plus room_name (the media room), sip_call_id, recording ids when started, and flags such as amd_enabled / tts_blast you can record without re-parsing your own request.

​
5. What integrators branch on

  • transcript.partial vs transcript.final. Streaming vendors emit interim hypotheses and committed finals; batch fallbacks emit only finals. Branch rendering and conversational memory on finals; use partials for live typing indicators at most. Never treat an interim transcript as the caller’s utterance.
  • audio.flush causes. The flush event’s cause field (barge_in / cancel / takeover / session_end) tells you which interrupt class happened — drive your UI and analytics off it instead of inferring from audio stops.
  • Failover observability. The failover happens in the gateway’s provider chain; your visible signal is on the event stream — an STT/TTS upstream error event, then a recovered session (pre-stream) or an ended one (mid-stream). If a session opens in a language Deepgram does not cover well, expect the chain to start one tier down by design (see per-language re-partitioning), not by failure.
  • Session resume. If the WebSocket drops within the resume window, reconnect with session.resume; the gateway replays what you missed and the agent picks up in place.

​
Worked example: one inbound turn with a Deepgram → Whisper boot failover

A caller dials a DID routed to your agent. The session has already been created by the inbound bridge; your client connects and the first Deepgram stream fails before any transcript — the chain advances to Whisper and replays the buffered audio so the caller’s opening words survive:
From your client’s perspective nothing about the failover is visible on the wire except a later-than-usual first partial. Inside the gateway, the Deepgram leg failed pre-stream, the preroll buffer replayed into Whisper, the preroll sealed on Whisper’s first transcript, and the rest of the session runs on the fallback with the same session options. Had Deepgram failed after the first partial, you would instead have seen an error event with OVP_STT_UPSTREAM_FAILED and no failover — that is the duplicate-transcript trade-off from section 3 showing up on the wire. The wire format itself (event envelopes, binary frame header, resume semantics) is specified in the Orbit Voice Protocol.

​
Cross-references