Lowest Latency Calls

Architecture, configuration, and reference for the OpenAI Realtime API + Twilio phone system.

To PLACE calls, manage prospects, and run campaigns: pair this skill with your own outbound dialer / campaign layer. This skill is about the real-time call infrastructure itself.

DO NOT CHANGE (confirmed working, breaks if altered)

The call flow, session config format, and audio path below were debugged through many iterations. Do not restructure without reading this entire skill.

Session config — FLAT format only

// CORRECT:
{ type: "session.update", session: {
    modalities: ["text", "audio"], voice: "cedar",
    turn_detection: { type: "semantic_vad", eagerness: "high", create_response: true, interrupt_response: true },
    input_audio_format: "g711_ulaw", output_audio_format: "g711_ulaw",
}}
// WRONG (API rejects): session: { type: "realtime", audio: { input: { format: ... } } }

Outbound call flow — caller-first

Callee picks up, says hello, THEN the agent responds. No forced greeting. Semantic VAD with create_response: true handles it automatically.

Audio path — direct passthrough

Audio deltas from OpenAI are already base64 g711_ulaw. Forward directly to Twilio. No PCM conversion, no gain control, no resampling.

Greeting trigger

conversation.item.create (user message) + response.create. NOT response.create with instructions. Trigger on session.updated, NOT session.created.

Twilio webhook

Must point to /twiml. Verify: check Twilio API, not assumptions.

SAFE TO TUNE

• Prompt size: smaller = faster inference. Reference outbound prompt is ~478 tokens.
• VAD eagerness: "high" first turn, "medium" after. Configurable.
• Tool loading: lean tools first turn, full set after first response.done.
• Voice: cedar is a solid default for all scenarios. Can change per scenario.
• Inference priming: text-only response.create during pre-warm warms pipeline without audio.
• Twilio edge: configure to colocate with your deployment region and OpenAI region for lowest RTT.

Debugging Checklist

Before adding patches when calls fail:

1. Is the websocket server process running? (systemctl status \x3Cyour-service>, pm2 status, or your equivalent)
2. Single owner on the websocket port? lsof -i :\x3CPORT>
3. Twilio webhook URL correct? Check the Twilio API, not local config files.
4. Check your server log (whatever path you configured — stdout, file, or journald)
5. OpenAI outage? Check status.openai.com
6. Session config accepted? Look for session.updated in logs. error after session.created = wrong config format.

Do not pile patches. If it worked before and doesn't now, check infrastructure first.

Restart Procedure (pattern)

Whatever process supervisor you use, the correct sequence is:

stop the websocket server
→ kill any orphaned listeners on the websocket port (lsof -i :\x3CPORT> -t | xargs kill)
→ start the websocket server

Always stop → kill orphans → start. A bare restart can leave a stale listener holding the port.

Restore from Snapshot (pattern)

Keep a known-good copy of sessionManager.ts (the file most affected by tuning) in a snapshots directory alongside the source. To restore:

copy snapshots/sessionManager-TUNED-\x3Cdate>.ts → src/sessionManager.ts
restart using the procedure above

Key Files (relative to the websocket-server project)

Reference Documents

All reference docs in {baseDir}/docs/:

Load the relevant doc before answering architecture or config questions.

Key Facts (always available without file load)

• Model: gpt-realtime-1.5 (flagship), gpt-realtime-mini (cost-sensitive)
• WebSocket: wss://api.openai.com/v1/realtime?model=gpt-realtime-1.5
• Audio: mu-law / PCMU at 8 kHz mono, base64 encoded
• Turn detection: semantic_vad with eagerness: "high" is the tested default
• Pre-warm timeout: 10 seconds (fallback to cold connect)

Lessons

1. Session config: flat format only. Nested is rejected.
2. Trigger greeting on session.updated, not session.created.
3. Semantic VAD works without prior audio response.
4. Verify infrastructure before debugging behavior.
5. Audio is already PCMU. No conversion needed.
6. Prompt size directly affects per-turn latency.
7. When patches pile up: stop, read docs, rewrite from baseline.