> ## Content Index
> Fetch the complete content index at: https://cms.syntrigen.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Voice agents: the latency budget is the design
- URL: https://cms.syntrigen.com/voice-agents-latency-budget/
- Published: 2026-06-17T09:00:00.000Z
- Updated: 2026-08-20T12:48:37.000Z
- Description: In text, a two-second pause is imperceptible. In conversation it is the difference between a system that feels present and one people talk over. Almost every voice architecture decision follows from that.
- Author: Syntrigen
- Tags: Voice, Agent architecture, AI in production

Text and voice look like the same problem with a different interface. They are not. Text tolerates thinking time; conversation does not.

Human turn-taking runs on gaps of roughly a couple of hundred milliseconds. Past about 800 milliseconds a pause reads as hesitation or a dropped line, and people start speaking again — which produces overlap, which produces confusion, which is the failure mode users describe as "it kept interrupting me".

So the design starts from a budget, and every component takes a share of it.

## The chain, and where the time goes

Speech in, speech out, roughly: endpointing → transcription → model → tool calls → speech synthesis → playback.

The instructive part is that the model is usually not the largest contributor.

**Endpointing** — deciding the person has finished — is often the biggest and least-examined cost. A conservative voice-activity threshold waits for silence to be sure, and that wait is added to every single turn. Tuning it is uncomfortable: too eager and you cut people off mid-sentence, too patient and the whole system feels slow.

**Transcription** should be streaming, with interim results, so the model can begin on a partial. Batch transcription of a finished utterance adds its entire duration to the budget.

**The model** should stream tokens out, and synthesis should begin on the first clause rather than the complete response. Waiting for a full answer before speaking wastes the largest easily-recoverable chunk of time in the chain.

**Tool calls** are the unbounded term. A database read is fine. A third-party API that takes 1.5 seconds has consumed the entire conversational budget on its own, and no amount of model or speech optimisation compensates.

## What this forces

**Everything streams, or the budget is gone.** Streaming is not an optimisation here, it is the architecture. Any component that requires a complete input before producing output moves its full duration onto the critical path.

**Slow tools need spoken cover.** If a lookup takes two seconds, the system says "let me check that" and speaks while it waits. This is not a trick — it is what a person does, and it converts dead air into a natural turn.

**Barge-in must be first-class.** People interrupt. The system has to stop speaking immediately, discard the rest of the planned utterance, and treat the interruption as the new turn. Retrofitting this into a request-response design is painful; assuming it from the start is not.

**Context stays small.** Long histories cost time as well as money, and time is the constraint. Aggressive summarisation of earlier turns is often necessary rather than optional.

**The model is chosen for latency as much as capability.** A slightly weaker model that responds in 400ms frequently produces a better conversation than a stronger one at 1.2 seconds, because the conversational failure is more damaging than the occasional weaker answer.

## What to measure

Not average end-to-end latency. Measure each hop separately, at the 95th percentile, in production. The average hides the turns that ruined the call.

Then add two that are specific to voice: **barge-in rate**, which tells you how often people felt the need to interrupt, and **time to first audio**, which is what the user actually experiences as responsiveness — long before the full answer arrives.

## The design consequence

Voice is the one place where a system's architecture is directly audible. A design that is merely correct sounds wrong, and users describe it in emotional terms — awkward, robotic, rude — that map precisely onto specific milliseconds in the chain.

Which makes the budget the specification. Write it down before choosing components, and hold each one to its share.