Operations

This chapter covers running the gateway day to day: where the logs are, how to read the response-time line, what to monitor, how far one instance goes, what to back up, how to lock it down, and which tuning settings are safe to change.

Logs

The gateway writes to standard output and standard error only. It keeps no log files of its own, apart from logs/performance.csv when LOG_LATENCY=true (the logs/ directory must be writable by the service user).

Lines start with INFO:, WARN:, ERROR: or DEBUG:, and many carry an emoji as well, so grep on the words, not the start of the line.

journalctl -u vexyl-gateway --since "1 hour ago" | grep -E "WARN|ERROR|LOOP-LAG|starvation"

Lines worth knowing:

A punycode deprecation warning at start-up is normal and harmless.

At boot the gateway also logs whether the HTTP whitelist is on: HTTP API IP whitelist enabled or HTTP API IP whitelist disabled (all IPs allowed). Check that line after every configuration change.

Measuring response time

Latency logging is off by default. Switch it on:

LOG_LATENCY=true

Restart the gateway. From then on you get three things: a LATENCY line per turn in the log, a row per turn in logs/performance.csv, and a performance.latency block in the call-ended webhook (see Webhooks and billing).

The line looks like this:

LATENCY [req-N]: buffer: Xms | stt: Xms | llm: Xms | tts: X | send: Xms | response: Xms | total: Xms | txdepth: ...

To find the slow stage, compare the columns across a handful of turns rather than judging one. A single slow turn is usually the provider; the same column slow on every turn is configuration.

logs/performance.csv has the same columns as the LATENCY line, one row per turn, plus a request id and a timestamp. Open it in a spreadsheet and sort by response.

For quick totals without logging, GET /calls/stats and GET /cache/stats are described in Active calls and stats and TTS cache.

Health checks for monitoring

Poll GET /health every 30 to 60 seconds from your monitoring system. The route is described in Health and licence.

curl -s http://127.0.0.1:8081/health

Alert on:

If concurrentCalls.active is above zero while your PBX shows no calls, sessions are stale. Restart the gateway at a quiet moment. See the troubleshooting chapter (17-troubleshooting.md).

Under Docker the image has a built-in health check, so docker ps shows healthy or unhealthy.

Where sessions are kept

By default, sessions live in memory. They are lost on restart, which also cuts any call in progress. To keep metadata that you posted before a call across restarts, or to share it between several instances, set up Redis:

REDIS_HOST=127.0.0.1
REDIS_PORT=6379
REDIS_PASSWORD=
REDIS_DB=0
SESSION_TTL=3600

SESSION_TTL is in seconds. storage.storage_type in /health shows memory or redis, and storage.redis_connected shows whether the connection is up.

Capacity

One gateway process handles call audio on a single thread, so one instance is bound to about one processor core. On normal hardware the licence cap (concurrentCalls.max in /health) is reached before the processor is. Provider speed matters more to the caller's experience than raw processing power.

To grow beyond one licence or one core:

• Run several instances, each with its own licence, HTTP port, SIP port and RTP port range. The licence is per machine id: instances on one host share a machine id unless they are containers with distinct hostnames, so ask Vexyl how to license them.
• Put them behind your PBX and let the PBX spread calls across them (one trunk or route per instance).
• Keep the instances' RTP ranges apart from each other and from the PBX's own range (see Do not let the RTP ranges overlap).
• Allow for upstream bandwidth: every call sends audio to the PBX or carrier in real time, so a small office uplink can become the limit before the processor does.
• Use Redis (above) if instances must share pre-call metadata.

Watch concurrentCalls.utilization and the "rejected due to concurrent limit" log line to decide when to add an instance.

Upgrades and rollback

The full steps, including how to roll back, are in Upgrading for the binary and Upgrade for Docker. The short checklist:

1. Announce a short window to the people who rely on the line.
2. Wait until concurrentCalls.active is 0. A restart cuts calls in progress.
3. Verify the checksum of the new download.
4. Swap in the new binary or image, keeping the old one for rollback.
5. Check GET /health and place one test call.
6. Watch the LATENCY lines for the first ten minutes and compare them with what you saw before.

Backups

Security checklist

The full picture, with what each listener trusts and what data is kept, is in Security.

• Keep the HTTP API on loopback, or restrict it with HTTP_ALLOWED_IPS. Never expose the HTTP port to the public internet. See Access.
• Put the browser WebSocket endpoint behind a TLS proxy, keep its own port on loopback, and set both WEBSOCKET_AUDIO_API_KEY and WEBSOCKET_AUDIO_ALLOWED_ORIGINS. See Secure it.
• Set SIP_ALLOWED_IPS to the exact addresses of your PBX or carrier. Open the RTP port range in the firewall to those addresses only.
• Make .env readable by the service user alone (mode 600).
• Set WEBHOOK_SECRET and BILLING_CDR_SECRET, and verify the signature on your receiver.
• Leave WEBHOOK_SSRF_CHECK on unless your receiver is on a private network.
• Rotate provider keys when staff who knew them leave.
• The container runs as user id 1000; make mounted directories writable for that user and no one else.

chmod 600 .env
ls -l .env

Tuning that is safe to touch

The defaults suit most installations. Change one setting at a time, restart, and compare LATENCY lines before and after.

RTP_PACER_LEAD_MS=500