LLM providers

An LLM provider is the service that writes a bot's replies. LiteBot ships with seven: Groq, OpenRouter, OpenAI, Gemini through AI Studio, Gemini through Vertex AI, Sarvam and Ollama. This chapter covers the variables each one reads, how a bot picks its provider and model, what happens when a provider fails, and how a tenant brings its own key. Read it before you set keys in .env, and again when replies start failing or coming from an unexpected model.

Realtime voice over /voice talks to Gemini Live directly and does not go through these providers; see Voice.

Choosing

A provider registers at startup only when its key (or, for Vertex and Ollama, its settings) is present. The log lists what registered:

Initialized 3 LLM provider(s): openrouter, gemini, gemini-vertex (default: gemini-vertex)

The gemini fallback model gemini-1.5-flash is an old model. Always set GEMINI_MODEL; .env.example sets models/gemini-2.5-flash. Keep the models/ prefix as shown there.

Every provider supports both plain and streaming replies, so any of them works with "stream": true on the chat API (Chat API).

How a bot picks its provider and model

Each bot names a provider and, optionally, a model (Bot configuration reference).

{ "id": "support", "provider": "gemini-vertex", "model": "models/gemini-2.5-flash" }

• provider defaults to groq when the bot omits it. If Groq is not registered on your server, the bot always runs on a substitute, so set provider on every bot.
• model is optional. Without it, the provider's default model answers.
• temperature (default 0.7) and maxTokens (default 1000) are passed with every request.

For each turn LiteBot resolves the provider in this order:

1. The bot's tenant has its own key for that provider: a tenant-owned instance answers (Per-tenant provider keys).
2. The named provider is registered and usable: it answers.
3. Otherwise DEFAULT_LLM_PROVIDER, if it is set, registered and usable.
4. Otherwise the first usable provider in registration order: groq, openrouter, gemini, gemini-vertex, openai, sarvam, ollama.
5. If nothing is usable, the first registered provider is tried anyway, with the log line "All providers unhealthy, attempting with first available provider".

Model name on a substitute

When a different provider answers than the one the bot asked for, LiteBot drops the bot's model (and any Gemini cache handle) so the substitute answers with its own default model. Sending models/gemini-2.5-flash to OpenRouter would fail with a 400. The log shows:

Provider gemini unavailable; openrouter serving with its default model

Set each provider's *_MODEL variable to a model you are happy to be served by, because that is what callers get during an outage. Analytics record the provider that actually answered, not the one requested.

Groq, OpenRouter, OpenAI, Sarvam

Gemini (AI Studio)

These behaviours apply to both Gemini transports.

• Context cache. For a bot with a system prompt, LiteBot caches the system prompt and tools on Google's side and reuses the cache on later turns. The first turn that misses starts creating the cache in the background and runs uncached. Prompts estimated below GEMINI_CACHE_MIN_TOKENS (about four characters per token) are never cached, because Google rejects small caches.
• Hedged requests. Gemini Flash usually answers in about a second but sometimes stalls for many seconds. Hedging fires a duplicate after the delay. Keep the delay well above your normal reply time, or you pay for two requests on healthy turns.
• Safety settings. All four harm categories are set to BLOCK_ONLY_HIGH. The stricter medium setting blocked ordinary Malayalam healthcare speech. A blocked reply is reported with finish reason content_filter. This is not configurable.
• Thinking. Gemini 2.x Flash runs with a thinking budget of 0 and Gemini 3 Flash with thinking level minimal, to keep first-token latency low. Text chat has no setting to change this; VOICE_THINKING_BUDGET applies only to realtime voice.
• Thought signatures. Gemini 3 models sign each tool call, and the signature must be sent back when the call is replayed in history or the request fails with a 400. LiteBot stores and replays it. For tool calls in old history without one (or made by another provider), it sends Google's documented bypass value so the turn still works.

Gemini on Vertex AI

gemini-vertex serves the same Gemini models from a Google Cloud region, authenticated with a service account instead of an API key. Bots opt in with "provider": "gemini-vertex". Measured from Mumbai, first token arrives in about 0.7 s on asia-south1 against about 1.2 s through AI Studio.

VERTEX_PROJECT_ID=my-gcp-project
VERTEX_LOCATION=asia-south1
VERTEX_CREDENTIALS_FILE=/opt/litebot/vertex-sa.json
VERTEX_MODEL=models/gemini-2.5-flash
DEFAULT_LLM_PROVIDER=gemini-vertex

Before you start: enable the Vertex AI API on the project, create a service account with the Vertex AI User role (roles/aiplatform.user), download its JSON key, and make it readable by the user LiteBot runs as. The file must be a service_account key with client_email and private_key.

Setting only one of the two required values logs "gemini-vertex needs both VERTEX_PROJECT_ID and VERTEX_CREDENTIALS_FILE" and skips the provider. An unreadable or invalid key logs "gemini-vertex not registered" and leaves the other providers running.

Region to global fallback

Regions do not serve every model, and new models reach regions weeks after global. When the configured region returns 404 for a model, LiteBot remembers that model as global-only for the life of the process, retries once on the global endpoint, and logs "Vertex: model not served in the configured region; using the global endpoint for it". The first turn on such a model pays the extra round trip; later turns go straight to global. A restart forgets the mapping.

Context caching works on Vertex too. Knowledge-base embeddings follow the same Vertex settings automatically (Knowledge base).

Ollama

LiteBot uses Ollama's native /api/chat endpoint. Ollama cannot take a per-tenant key.

Timeouts and the circuit breaker

Each provider call runs inside a circuit breaker. After a run of failures the breaker opens and rejects calls for a minute, then lets a probe through; two successful probes close it again.

Known gap: .env.example sets CIRCUIT_BREAKER_TIMEOUT=30000, which silently overrides LLM_REQUEST_TIMEOUT, the per-provider timeouts and every tool's metadata.timeout. Remove it if you rely on those. A breaker is created on first use with the settings of that moment, so timeout changes need a restart.

Failover within a request

With the breaker on and DISABLE_PROVIDER_FALLBACK not true, a failed or rejected call is retried once on another provider: DEFAULT_LLM_PROVIDER first, then registration order, skipping the one that failed. The bot's model is dropped so the substitute uses its default. There is no further fallback after that one retry. Streaming calls never fail over within the request; they rely on health marking for the next turn.

Health and recovery

Separately from the breaker, LiteBot tracks each provider's health.

• Three consecutive failed calls mark a provider unhealthy, with "Provider gemini marked unhealthy after 3 failures" in the log. Any success resets the count.
• A stream cut off mid-reply is treated as a network blip and does not count as a failure.
• An unhealthy provider is skipped when choosing a provider for new turns, so bots that name it move to a substitute.
• After PROVIDER_RETRY_MS (default 60000) one request is allowed through as a probe ("Probing unhealthy provider gemini"). Success makes the provider healthy again; failure starts a new window. Only one request probes per window.
• POST /api/bots/providers/health-check (admin) probes every provider immediately and updates health.

PROVIDER_RETRY_MS and the probe are present in the current working tree but not yet committed or listed in .env.example. Without them, an unhealthy provider stays out until a restart or a manual health check.

curl -s http://127.0.0.1:3345/api/bots/providers/status -H "Authorization: Bearer <token>"
curl -s -X POST http://127.0.0.1:3345/api/bots/providers/health-check -H "Authorization: Bearer <token>"

{ "success": true, "data": { "available": ["gemini", "gemini-vertex"], "total": 2, "healthy": 1,
  "providers": { "gemini": { "healthy": false, "consecutiveFailures": 3, "lastError": "Request failed with status code 402" } } } }

Responses to requests without tools are cached in Redis for 10 minutes when Redis is connected. The cache key includes the bot, provider, model, system prompt and messages.

Per-tenant provider keys

A tenant can bring its own key for groq, openrouter, gemini, openai or sarvam. Bots of that tenant then run on the tenant's key; other tenants keep using the global keys. gemini-vertex and ollama cannot take a tenant key.

curl -s -X PUT http://127.0.0.1:3345/api/tenants/acme/providers/gemini \
  -H "Authorization: Bearer <token>" -H "Content-Type: application/json" \
  -d '{"apiKey":"<tenant-gemini-key>","model":"models/gemini-2.5-flash"}'

{ "success": true, "data": { "provider": "gemini", "keyHint": "x9Qz", "model": "models/gemini-2.5-flash" } }

• GET /api/tenants/:id/providers lists the providers with the last four characters of each key only. DELETE /api/tenants/:id/providers/:provider removes a key.
• Keys are encrypted at rest with a key derived from JWT_SECRET. If you change JWT_SECRET, stored keys can no longer be decrypted; LiteBot logs a warning and the tenant's bots fall back to the global key.
• A tenant key that fails is never replaced by the global key: the error surfaces, the tenant instance is outside health marking and failover, and it has its own breaker.
• model is optional; without it the global default model for that provider is used.
• Setting or removing a key takes effect at once on the node that received the request. Another node that had no key for that tenant notices a new one within a minute; a node that already built an instance with the old key keeps using it until it restarts.

Where to go next

• Set a bot's provider and model: Bot configuration reference
• Give the model tools to call: Tools
• Tenants and who may set keys: Users, roles and tenants
• What each model costs and how fast it answers: Cost and performance
• Replies failing or coming from the wrong model: Troubleshooting