Article

How OpenAI delivers low-latency voice

Author

Oleksandr Kotliarov

Date

July 19, 2026

Reading Time

9 min

A spoken reply feels instant to a person at somewhere under a second. Cross that line and the conversation stops feeling like a conversation — the other side sounds like it is buffering, and you start talking over it. OpenAI’s Realtime API sits under that line, and the interesting part is how. It is not a faster model or a bigger GPU. It is a set of architectural decisions that treat latency as a property of the system’s shape, not a number you tune at the end.

That distinction is the whole point of reading the design. Most teams reach for latency wins in the wrong place — a smaller model, a warmer cache, a closer region — after the architecture has already set a floor they cannot get under. Voice is where that floor is audible, which makes it a good place to watch the moves that matter.

Here is the pipeline, in our own words, from the microphone to the speaker — and then the four moves worth stealing for anything real-time you build.

The old pipeline had three seams

The obvious way to build a talking model is to chain three models you already have. Speech-to-text turns the user’s audio into words. A language model reads the words and writes a reply. Text-to-speech turns the reply back into audio. It works, it is easy to reason about, and every seam between the boxes is a place latency collects.

Each stage in that chain traditionally waits for the one before it to finish. The transcriber returns the full transcript, then the language model starts. The language model returns the full reply, then speech synthesis starts. Three complete-before-forward handoffs, stacked. Even with fast components, the handoffs alone add up: the comparison Deepgram draws between cascaded and end-to-end speech systems puts the cascaded overhead in the hundreds of milliseconds before either the model or the network has done anything interesting. The user hears the sum.

The fix is not to make the three boxes faster. It is to stop having three boxes.

Move one: collapse the stages

OpenAI’s Realtime models — gpt-realtime and the smaller gpt-realtime-mini in the current docs — are natively multimodal. Audio goes in and audio comes out of a single model. There is no transcript handed to a second system and no text handed to a third. The two internal seams where latency used to collect are simply gone, because the boundary they sat on no longer exists.

Before-and-after diagram titled "The handoffs were the latency." Top row: three chained grey boxes — speech-to-text, language model, text-to-speech — with two burnt-orange "latency seam" markers in the gaps between them. Bottom row: a single black box labelled "one model, speech-to-speech" with no seams at all.

This is the move with the largest payoff and it is the one teams resist, because a single fused stage is harder to build, harder to debug, and harder to swap a component out of. A cascade is legible: you can point at the transcriber when it mishears. A speech-to-speech model is one box you cannot open. That legibility is exactly what you are paying the latency tax for. When the tax is a person deciding your product feels broken, the trade tilts.

The general form: any sequential handoff between services is a latency floor you set by drawing the boundary there. Before you optimize the stages, ask which boundaries have to exist at all. The cheapest round trip is the one you deleted.

Move two: stream everything, emit before you finish

Collapsing the stages removes the seams between them. Streaming removes the wait inside each one.

The instinct in most software is to compute a complete result and then forward it. That is a latency cliff. The Realtime path inverts it: the model begins emitting response audio while the user’s turn is still being processed, and the client begins playing that audio while the model is still generating the rest of it. Nothing waits for a whole anything.

The published pipeline breakdowns give a feel for the cadence — Retell’s walk-through of real-time voice describes partial transcripts landing every few tens of milliseconds, language tokens streaming at dozens per second, and audio rendered in chunks a few hundred milliseconds long, each stage starting on partial input from the last. The exact numbers vary by vendor and setup; treat them as illustrative. The shape is the point: the first audio byte leaves long before the last is decided.

Streaming is not a feature you add to a finished system. It is a constraint you accept at the start, and it reshapes everything downstream. Your components have to produce partial results. Your error handling has to cope with a reply that was half-spoken when it turned out wrong. Your buffering and your cancellation logic have to assume the output is already in flight. Teams that bolt streaming on late discover these are not small changes — they are the architecture.

Move three: make turn-taking an inference problem

A voice system has to know when the human has stopped talking. Get it wrong in one direction and it interrupts a person mid-thought; get it wrong in the other and it sits in dead air after they have finished. This is the part that separates a demo from something you can actually talk to.

The naive method is silence detection: wait for the audio to go quiet for some threshold, then treat the turn as over. It is cheap and it is wrong constantly, because people pause mid-sentence — to think, to breathe, to find a word — and a silence timer reads every one of those as “your turn.” Push the threshold up to stop the false interruptions and now the system feels sluggish, because it waits out every real pause too.

OpenAI’s Realtime API exposes turn detection as a configurable behavior in the VAD documentation, including a semantic mode that decides a turn is complete from what was said, not just from how long it has been quiet. An eagerness setting trades responsiveness against patience. The distinction is that this is a learned judgment about content, not a signal-processing threshold on amplitude. Whether the sentence sounds finished is a question the model is already equipped to answer, so turn-taking becomes another inference rather than a timer bolted to the side.

Diagram titled "Silence is the wrong signal." One spoken utterance with a mid-sentence pause is read three ways. The silence timer fires a burnt-orange "false turn-end" at the pause and drops the rest of the sentence; semantic detection waits and ends the turn only when the sentence is actually complete.

The same machinery runs during the model’s own output, which is what makes barge-in work. Detection does not pause while the assistant is speaking; it keeps listening, and when the user starts talking over the reply, the in-flight response is cancelled and the new input takes over. If you have ever interrupted a voice assistant and had it keep talking for three seconds, you have heard a system that stopped listening the moment it started speaking. Continuous detection during output is not a nicety. It is the difference between a conversation and a voicemail.

Move four: push transport to the edge

The last stretch is the network, and for a browser talking to a model it is the part teams most often get wrong by defaulting to whatever is easy.

The easy default is a WebSocket, which runs over TCP. TCP guarantees ordered, complete delivery, and that guarantee is the problem: a single lost packet stalls everything behind it until it is retransmitted, because TCP will not hand your application packet six until packet five has arrived. For a file that is correct. For live audio it is a stutter, and on a real network — mobile, contended Wi-Fi, anything lossy — the stutters are constant.

OpenAI runs the browser edge on WebRTC over UDP instead. UDP does not wait. A lost packet is a small gap the audio codec papers over, not a stall that blocks the stream, and a jitter buffer on the receiving side absorbs the variation in arrival times. The result is a latency floor well under what a proxied TCP connection can hold on the same network. This is why the browser path is WebRTC and not the WebSocket you would reach for first.

The ByteByteGo teardown of the server side is where the edge thinking gets concrete. Incoming connections hit a stateless relay placed geographically close to the user, whose only job is to route packets onward — and it routes them by reading the ICE username fragment already in the packet, with no database lookup on the hot path. The actual WebRTC state, the encryption and session handshakes, lives on a separate stateful transceiver that the relay forwards to. Splitting the two lets the expensive stateful part sit wherever it needs to while the cheap stateless part sits next to the user. Even the socket handling is tuned for it: the writeup describes SO_REUSEPORT to spread load across threads and pinning goroutines to OS threads to keep lock contention and garbage-collection pauses out of the packet path.

Diagram titled "Route at the edge, keep state central." A path runs from the user's browser to a burnt-orange stateless relay placed close to the user, which routes packets by reading them directly with no lookup, then forwards to a central stateful transceiver that holds the WebRTC session state.

None of that is voice-specific. It is the standard shape of low-latency networked systems — do the cheap routing at the edge, keep the expensive state central, and keep both the hot path and the runtime out of each other’s way. Voice just makes the payoff audible.

What to take to your own system

You are probably not building a speech-to-speech model. The moves still transfer, because none of them are about voice. They are about where latency lives in any system that has to feel immediate.

Count your handoffs before you tune your stages. Every complete-before-forward boundary between services is a floor you set by drawing it there. The largest win is usually a seam you delete, not a stage you speed up.
Decide streaming on day one. Emitting partial results changes your error handling, buffering, and cancellation, so it is not a late addition. If the experience has to feel instant, the components have to produce output before they are done.
Turn heuristics into inference where you can. The silence-threshold problem — a cheap rule that is wrong at the edges — shows up all over real-time systems. A learned judgment on the actual content often beats a threshold on a proxy signal.
Match transport to the workload, not to the default. Reliable, ordered delivery is the right choice for a file and the wrong one for a live stream. Push the cheap routing to the edge and keep the expensive state where it belongs.

Latency is not a setting you find at the end. It is decided when you choose the shape of the system — how many boundaries it has, whether its stages stream, where its state lives. OpenAI’s voice stack is a clean read on all four decisions going the right way. The reason it feels instant is that the instantaneity was designed in, not tuned in afterward.