Skip to content
effect-uai

Realtime

For voice assistants you can ship today, see Speech → Voice loop. The composed STT → LLM → TTS pipeline covers the common case and runs on the shipped speech primitives.

This page is about the other archetype: one long-lived duplex session where the model owns turn-taking, can interrupt itself on detected user speech, and can take camera frames alongside audio. That primitive isn’t shipped yet.

What Realtime adds

A pipeline of Transcriber → LanguageModel → SpeechSynthesizer gives you most of a voice agent — but it has four properties a native duplex API will improve on:

  • Model-native barge-in. Voice loop’s interrupt fires on stop-words detected client-side from finals. A native session lets the model decide it’s been interrupted (server-side VAD on the input audio) and trim its own response — no keyword list, no client detection.
  • Mid-utterance tool calls. In the pipeline, a turn is atomic: STT → LLM → TTS, then the next turn. A native session can interleave tool calls into the same continuous audio stream.
  • Sub-200 ms turn-taking. The pipeline pays for every boundary (STT-final, LLM TTFT, TTS first-byte). A native session amortizes some of that overhead by keeping a single WebSocket open.
  • Camera-in streams. When a provider ships realtime vision, pointing a phone camera at something and getting a spoken answer becomes one session — not a pipeline of “snapshot every N seconds → multimodal LM → TTS.”

If you don’t need any of those, the voice-loop pipeline is the simpler answer and exercises the same primitives you already use.

Coming soon

When this lands, @effect-uai/core will ship a RealtimeSession service. Likely shape: a session value carrying a Queue<RealtimeInput> you push into (audio frames, video frames, text, control) and a Stream<RealtimeEvent> you consume from (audio chunks, transcript deltas, tool calls, turn boundaries).

Provider candidates:

  • OpenAI Realtime — WebSocket and WebRTC transports, gpt-realtime family. Audio in / audio out today; vision input on the roadmap.
  • Google Gemini Live — WebSocket, audio + video in / audio + text out. The closest thing to “point your camera, get an answer” today.

The right primitives — backpressure, cancellation, interrupt semantics — get designed alongside the first integration, not in advance.

See also

  • Voice loop — the ship-today voice agent.
  • Speech — the one-direction primitives the voice loop is built from.