Voice agents: the latency budget is the design

Text and voice look like the same problem with a different interface. They are not. Text tolerates thinking time; conversation does not.

Human turn-taking runs on gaps of roughly a couple of hundred milliseconds. Past about 800 milliseconds a pause reads as hesitation or a dropped line, and people start speaking again — which produces overlap, which produces confusion, which is the failure mode users describe as "it kept interrupting me".

So the design starts from a budget, and every component takes a share of it.

The chain, and where the time goes

Speech in, speech out, roughly: endpointing → transcription → model → tool calls → speech synthesis → playback.

The instructive part is that the model is usually not the largest contributor.

Endpointing — deciding the person has finished — is often the biggest and least-examined cost. A conservative voice-activity threshold waits for silence to be sure, and that wait is added to every single turn. Tuning it is uncomfortable: too eager and you cut people off mid-sentence, too patient and the whole system feels slow.

Transcription should be streaming, with interim results, so the model can begin on a partial. Batch transcription of a finished utterance adds its entire duration to the budget.

The model should stream tokens out, and synthesis should begin on the first clause rather than the complete response. Waiting for a full answer before speaking wastes the largest easily-recoverable chunk of time in the chain.

Tool calls are the unbounded term. A database read is fine. A third-party API that takes 1.5 seconds has consumed the entire conversational budget on its own, and no amount of model or speech optimisation compensates.

What this forces

Everything streams, or the budget is gone. Streaming is not an optimisation here, it is the architecture. Any component that requires a complete input before producing output moves its full duration onto the critical path.

Slow tools need spoken cover. If a lookup takes two seconds, the system says "let me check that" and speaks while it waits. This is not a trick — it is what a person does, and it converts dead air into a natural turn.

Barge-in must be first-class. People interrupt. The system has to stop speaking immediately, discard the rest of the planned utterance, and treat the interruption as the new turn. Retrofitting this into a request-response design is painful; assuming it from the start is not.

Context stays small. Long histories cost time as well as money, and time is the constraint. Aggressive summarisation of earlier turns is often necessary rather than optional.

The model is chosen for latency as much as capability. A slightly weaker model that responds in 400ms frequently produces a better conversation than a stronger one at 1.2 seconds, because the conversational failure is more damaging than the occasional weaker answer.

What to measure

Not average end-to-end latency. Measure each hop separately, at the 95th percentile, in production. The average hides the turns that ruined the call.

Then add two that are specific to voice: barge-in rate, which tells you how often people felt the need to interrupt, and time to first audio, which is what the user actually experiences as responsiveness — long before the full answer arrives.

The design consequence

Voice is the one place where a system's architecture is directly audible. A design that is merely correct sounds wrong, and users describe it in emotional terms — awkward, robotic, rude — that map precisely onto specific milliseconds in the chain.

Which makes the budget the specification. Write it down before choosing components, and hold each one to its share.