LiteBot

Cost and performance

This chapter explains how LiteBot puts a price on every LLM call and session, where you see those figures, what response times the framework is built for, and which settings make a bot cheaper or faster. Read it before you choose a provider and model for a production bot, and again when a bill or a slow bot needs explaining. The stored records it refers to are described in Conversations and analytics.

All LiteBot cost figures are in US dollars and are estimates from list prices built into the release. Your provider's invoice is the real figure.

How cost is computed

Every LLM call is priced once, when it finishes, and the result is stored with the call. LiteBot takes the first rule that applies:

Order Rule Applies to
1 Voice calls are priced per modality: text and audio, in and out, at the Live model's rates. Gemini Live voice sessions
2 The provider reported the cost it actually billed. That figure is used as is. OpenRouter (LiteBot asks it for usage accounting)
3 The provider computed the cost itself, counting cached tokens at the cache rate. That figure is used. Gemini and Gemini on Vertex
4 Tokens times the built-in price table, per million tokens. Cached tokens are charged at the model's cached rate, or a quarter of the input rate when the table has none. Groq, OpenAI, Sarvam, Ollama, and any call where the above give no figure

What happens when a price is missing:

  • Unknown Gemini model: priced at a fallback of $1.00 input and $1.50 output per million tokens, and a warning No Gemini pricing for model ... is logged once. The fallback over-reports rather than under-reports.
  • Unknown model on another provider: priced at the first model in that provider's table, with a warning in the log.
  • Unknown provider, or no token counts: cost 0.
  • gemini-vertex uses the same prices as gemini. Ollama is always 0. The Sarvam prices in the table are placeholders; check your Sarvam invoice.

LiteBot prices tokens only. It records no charge for storing a Gemini context cache, and no cost for embeddings, vector search or your own servers.

Built-in Gemini prices

USD per million tokens, as built into this release. Unlisted Gemini models get the fallback above.

Model Input Cached input Output
gemini-2.5-flash-lite 0.10 0.01 0.40
gemini-2.5-flash 0.30 0.03 2.50
gemini-2.5-pro 1.25 0.125 10.00
gemini-3.1-flash-lite 0.25 0.025 1.50
gemini-3.5-flash-lite 0.30 0.03 2.50
gemini-3.5-flash 1.50 0.15 9.00
gemini-3.6-flash, -3.7-flash, -3.8-flash 0.75 0.075 3.75
gemini-3.1-pro-preview 2.00 0.20 12.00

The 3.6, 3.7 and 3.8 flash prices are promotional until 2026-12-31. From 2027-01-01 they double, and a release that still carries the old figures will under-report.

Voice (Gemini Live), USD per million tokens:

Model Text in Audio in Text out Audio out
gemini-3.1-flash-live-preview 0.75 3.00 4.50 12.00
gemini-2.5-flash-native-audio-preview-12-2025 0.50 3.00 2.00 12.00

Thinking tokens are charged at the text-out rate. A Live model not in this table costs 0 and logs No Live pricing data for model.

A worked example

A chat turn on gemini-2.5-flash with a 6,000-token prompt (system prompt, tools and history), of which 5,000 come from the context cache, and a 100-token answer:

Part Tokens Rate Cost (USD)
Uncached input 1,000 0.30 0.00030
Cached input 5,000 0.03 0.00015
Output 100 2.50 0.00025
Turn 0.00070

Without the cache the same turn costs 0.00205, about three times as much. A turn that calls a tool makes a second LLM call that re-reads the prompt, so it costs roughly twice a plain turn.

Where cost shows

Where What
POST /api/chat reply, usage Tokens for the whole turn (all LLM calls summed) and cost in USD where the provider computes one (Chat API).
request_analytics cost and costData per LLM call; costData.costSource is provider-reported or provider-computed when rule 2 or 3 applied.
session_analytics totalCost, averageCostPerMessage and token totals per session.
GET /api/analytics/report Cost per bot, per day, and split into voice and chat.
GET /api/conversations/:sessionId The session's cost and tokens in its analytics block.
Dashboard, Analytics Usage and Reports tabs (Dashboard).

To compare with an invoice, sum request_analytics.cost for the provider over the invoice period. A large gap usually means a model missing from the price table (check the log for the warnings above) or prices that changed after the release.

Performance targets

LiteBot is designed for these figures. They are goals for a correctly sized server, not guarantees.

Measure Target
API response time Under 500 ms at the 95th percentile
Vector search Under 100 ms
Tool execution Under 200 ms on average
Memory Under 2 GB per bot instance
Load 100 or more concurrent conversations

In practice the LLM is most of a turn's time. A one-sentence reply is mainly the model's time to first token, and each tool round-trip adds one more LLM call.

Making replies faster

Choose the provider endpoint near you

For Gemini, the route matters as much as the model. Measured from Mumbai on 2026-09-08 with gemini-2.5-flash, thinking off, time to the first streamed chunk:

Endpoint First chunk
AI Studio (gemini) 1.1–1.3 s
Vertex asia-south1 (gemini-vertex) 0.63–0.74 s
Vertex asia-southeast1 0.80–0.92 s
Vertex global 0.88–1.35 s
Vertex us-central1 2.1–2.5 s

AI Studio requests from India are served from North America. For deployments in India, register gemini-vertex with VERTEX_PROJECT_ID, VERTEX_LOCATION=asia-south1 and a service-account key, and set the bot's provider to gemini-vertex (LLM providers). Regions serve fewer models than global: a model the region does not serve falls back to global and runs at global speed. The first call each hour carries a token exchange of about 400 ms.

Keep thinking off

LiteBot sends thinkingBudget: 0 to Gemini 2.x flash models and thinkingLevel: minimal to Gemini 3.x flash models by default. Turning thinking on adds seconds to the first token. Voice sessions use VOICE_THINKING_BUDGET (default 0).

Stream

Send "stream": true to /api/chat so a voice gateway can start speaking on the first sentence. For bots with tools, STREAM_HOLD_TOOL_TURNS (on by default) holds back text until the call that ends the turn; turning it off gives earlier audio but lets callers hear interim text a tool result then corrects (Environment reference, part 2).

Cut LLM round-trips

  • Write tools so one call answers the question, instead of chains of lookups. Each extra round-trip is a full LLM call in time and cost.
  • Keep REACT_MAX_ITERATIONS low (default 5).
  • Look in the traces for turns with several llm_followup steps (Request traces).

Guard against stalls

GEMINI_HEDGE_DELAY_MS (default 6000) sends a second identical request when no first token has arrived in time and keeps the faster answer. Keep it well above your normal first-token time: each hedge is paid twice.

Tool timeouts and caching

  • A slow webhook holds the whole turn. Set metadata.timeout on slow tools and keep TOOL_EXECUTION_TIMEOUT (default 5000 ms) for the rest.
  • Timeouts are enforced only with ENABLE_CIRCUIT_BREAKER=true. If CIRCUIT_BREAKER_TIMEOUT is set, it replaces every per-tool timeout; leave it unset to keep them.
  • Mark read-only tools metadata.cacheable: true with a metadata.cacheTTL, so repeat look-ups such as a department list come from memory (Tools).

Knowledge base

  • Turn on the search result cache (VECTOR_CACHE_ENABLED=true) and keep the start-up warm-up on, so frequent questions skip embedding and search.
  • With VERTEX_* set, Gemini embeddings go through the same Vertex region as chat, which avoids the North-American round-trip for every query embedding.
  • Keep documents focused. A smaller, cleaner collection gives faster searches and shorter, cheaper prompts (Knowledge base).

Many conversations at once

  • Above about 50 concurrent conversations, or with more than one LiteBot process, run Redis and set REDIS_WRITE_THROUGH=true so sessions are shared and MongoDB writes leave the request path.
  • ENABLE_MESSAGE_QUEUE=true moves conversation writes to background workers.
  • The API rate limit counts per IP. A voice gateway that sends every call from one address needs API_RATE_LIMIT sized for all its calls.
  • Voice sessions are capped by MAX_VOICE_SESSIONS_CONCURRENT (default 10 per process).

Making replies cheaper

  1. Leave context caching on. GEMINI_CACHE_ENABLED is on by default and caches each bot's system prompt and tools. Prompts shorter than GEMINI_CACHE_MIN_TOKENS (2,048) are not cached.
  2. Pick the smallest model that handles the bot. On the table above, gemini-2.5-flash-lite costs a third of gemini-2.5-flash for input.
  3. Shorten the prompt. The system prompt is re-read on every LLM call, including each tool round-trip. Facts the bot rarely needs belong in the knowledge base.
  4. Send less history. LLM_HISTORY_MAX_MESSAGES (default 40) sets how many messages go to the model each call.
  5. Turn failover off (DISABLE_PROVIDER_FALLBACK=true), so a failing provider does not send traffic, and its cost, to another provider with a model name it does not know.
  6. Watch voice costs separately. Audio output costs several times text output. The report's voice and chat split shows where the money goes.

Measuring latency

Source What it shows
metrics in a POST /api/chat reply Measured ms for the turn: totalTime, llmTime, toolsTime, toolBreakdown per tool, dbTime, formattingTime, and iterations.
request_traces duration of every LLM call and tool call in a turn.
GET /api/analytics/response-timeline Average, maximum and p95 LLM latency per hour or day, to place a slowdown in time.
GET /api/analytics/tool-performance Calls, failures and average duration per tool.
request_analytics.responseTime Per LLM call, for your own queries.

The LLM, database, tools and formatting split on the dashboard Performance tab is an estimate from the average (70, 10, 15 and 5 per cent), not a measurement. Use the sources above when you need real figures.

Where to go next

Environment (2 of 2)