Text to speech

Text to speech (TTS) turns the bot's reply into the audio the caller hears. This chapter lists every TTS provider the gateway supports, the variables each one reads, how to override the voice for a single call, and the cache and playback settings that shape what callers hear. Pick a provider, set its required key, and restart the gateway. Ready-made combinations are in Configuration recipes; the recogniser side is in Speech recognition.

Choosing

Set TTS_PROVIDER to one of the values below. The default is sarvam.

TTS_PROVIDER=sarvam

Streaming providers start playing while the sentence is still being rendered, so the caller hears audio sooner. Batch providers render the whole sentence first, so expect a longer pause before each reply. When a provider fails on a turn, the gateway falls back to another configured provider so the caller is not left in silence.

Costs are billed per provider and model with built-in rates. To change a rate, create config/pricing.json from the example in Webhooks and billing and edit it.

Sarvam

Model and speaker must match. v4-flash rejects a bare v3 name and v3 rejects a persona id. The gateway maps one to the other automatically (kavya becomes kavya_hi_conversational and back) and logs a warning once, but set the right form to keep logs quiet. v4-flash has no Malayalam or Odia persona, but it is multilingual: Malayalam text spoken through a Hindi persona is correct Malayalam. Speaker ids keep their own language tag, so a Hindi-tagged persona such as shilpa_hi_narration works for Malayalam.

With v4-flash set SARVAM_TTS_STREAM_TRANSPORT=http. The WebSocket transport cuts the tail off many sentences on that model; HTTP streaming has the same first-audio speed without the cut. Cached audio made over WebSocket carries the cut, so purge cached v4-flash audio after switching (see The TTS cache). If a stream fails, the gateway tries the other transport, then REST batch, then another provider.

Language codes are mapped to Sarvam's list before the request: en-US and en-GB become en-IN, and or-IN becomes od-IN. Cache keys and SARVAM_TTS_SPEAKER_<LANG> still use the code your session carries.

Google

Set voices per language, not globally. Google rejects a voice whose language does not match the request, and the turn then plays silence. A single GOOGLE_TTS_VOICE_NAME applies to every language and silently breaks a multi-language bot, so use it only on a bot that speaks one language.

The request language is derived from the voice name: zh-CN uses cmn-CN voices and ar-AE uses ar-XA voices. A name that does not match its language is rejected.

Keep Chirp3-HD voice names. Other voice types drop back to REST batch and lose progressive streaming. All configured languages default to Chirp3-HD except Odia: Google publishes no Odia voice, so an Odia turn on Google falls to another configured provider.

Gemini

The 3.8 models run on Vertex AI, in location global only. The same service account is used. Cloud TTS and Vertex are separate backends, so a Cloud TTS-only model keeps its original request while a 3.8 model goes to Vertex. A per-call tts_model can send one call to Vertex while the rest of the host stays on Cloud TTS.

ElevenLabs

The first call after an idle period pays a socket handshake; warm turns do not. v3 is slower to first audio than flash, so use it only for languages flash cannot speak. v3 rejects the latency optimisation parameter, which the gateway leaves out for it.

Smallest

Most catalogue voices are multilingual, so one voice usually covers every language you serve. Streaming and batch return identical audio, so cached entries are interchangeable between them.

OpenRouter

One key reaches many voice vendors. Choose the vendor and model with OPENROUTER_TTS_MODEL. It is a batch path with no streaming, so replies start after synthesis finishes.

Audio arrives at a model-specific rate; the gateway reads it and resamples as needed. Listen to a voice before committing: a round trip through speech recognition proves the words are intelligible, not that they sound natural.

Deepgram

Built-in voices cover English (US, GB, India), German, Spanish, French, Italian, Japanese, Korean, Dutch, Polish, Portuguese (Brazil), Russian and Chinese. Indian-language callers are better served by another provider.

Azure

Murf

Murf has no streaming path: the whole sentence is rendered before playback. Dedicated voices cover English, Hindi, Tamil and Bengali; other Indian languages go through a multilingual voice.

Vexyl

A self-hosted Indian-language engine reached over a WebSocket, with named speakers and style control.

Per-call voice and model

A single call can use a different provider, model or voice from the host default. Set these keys in the session metadata before the call is answered (from the dialplan), as SIP headers on the INVITE, or in an outbound call request. Every accepted voice and model id is listed in Per-call settings reference. The metadata endpoints are in HTTP API.

curl -X POST http://127.0.0.1:8081/session/UUID/metadata \
  -H "Content-Type: application/json" \
  -d '{"tts_provider": "smallest", "tts_voice": "avni"}'

Only the provider named by tts_provider reads tts_model and tts_voice, so a fallback to another provider never receives a voice id it cannot use. Values are sanitised on entry: a model is vendor/name or a bare id, and a voice is at most 160 characters of letters, digits, and _ . : -. Browser widget sessions cannot set these keys. The cache key and the billing record follow the resolved model and voice.

Custom voices

Vexyl designs a Gemini voice from a text description, or replicates a voice from a consent-recorded sample, on request, and hands you a voice_<id>. Set it as GEMINI_TTS_VOICE_NAME or GEMINI_TTS_VOICE_<LANG>, or pass it per call as tts_voice. The gateway routes such calls to Vertex automatically, whatever the model. A description sets persona and timbre, not a regional dialect; only a replicated voice captures a real accent.

The TTS cache

The cache stores rendered audio on disk so a repeated sentence costs no synthesis and plays at once. It is off unless you enable it. It suits greetings, prompts, menus and FAQ answers; it does little for replies that are unique every time.

The cache key includes the text, language, provider, model and voice settings, so a changed voice makes new entries instead of serving the wrong audio. The old entries still sit on disk, though, and cached audio keeps the voice it was made with. Purge the cache after you change a voice, speaker, model or transport (for example moving Sarvam v4-flash from WebSocket to HTTP). Stop the gateway first, and start it again afterwards.

• Binary install: stop the service, run rm -rf cache/tts/* in the gateway's working directory, start the service.
• Docker with a bind mount: stop the container, run rm -rf ./cache/tts/* on the host, start it.
• Docker with a named volume: run the commands below.

docker compose down
docker run --rm -v PROJECT_vexyl-cache:/c busybox sh -c 'rm -rf /c/tts/*'
docker compose up -d

Check the hit rate:

curl -s http://127.0.0.1:8081/cache/stats

The response covers the time since the process started. hitRateChars is the figure that tracks your bill, because a hit on a three-character "Yes" and a miss on a long sentence are not an even trade. hitRateRequests counts requests instead. byProvider gives hits, misses, hit characters and miss characters per provider, and cache gives disk entries, size and the last cleanup. Per-call figures are in the billing record.

Pre-render phrases before callers need them:

curl -X POST http://127.0.0.1:8081/cache/warmup \
  -H "Content-Type: application/json" \
  -d '{"phrases": [{"text": "Hello, how can I help you?", "languageCode": "en-IN"}]}'

Warm-up renders through the same path live sentences use, so entries land under the key calls will read. The response reports cached (newly rendered), skipped (already cached), errors and total. When two calls need the same uncached sentence at once, they share one synthesis.

Each miss on the sentence-streaming path is logged as CACHE MISS (streaming) with the text, which is the list to mine when choosing what to warm up. Cache behaviour during outages is covered in Troubleshooting.

Sentence streaming and seams

When the bot streams its reply, the gateway cuts it into sentences, renders them in order and plays them back to back. These settings control how smooth that is.

Barge-in, interruptions and the thinking sound are described in Conversation behaviour; day-to-day checks are in Operations.