Why low voice AI latency can make calls worse

Say you run a pest control company in Sacramento. Six technicians, one office line, and a phone that rang out most afternoons until you bought an AI receptionist in July. It books jobs now. It also does something your old answering service never did: it starts talking while the caller is still reading out their gate code.

The dashboard reports voice AI latency of 740 milliseconds, which sounds quick. Your caller doesn't hear 740 milliseconds. They hear themselves cut off mid-number, repeat the digits, get an apology, repeat them again, then hang up and call the next company on the results page.

Voice AI latency is two problems pulling against each other, and one setting controls both. Shorten the silence your agent waits out before it decides a caller has finished and it feels quick, but it cuts people off mid-sentence; lengthen it and the agent is patient, but it sounds slow. The right number depends on what your callers actually say, not on a benchmark in a vendor's deck.

What voice AI latency actually measures

The number on the dashboard is usually model response time. The number your caller feels is the gap between the moment they stop talking and the moment they hear a voice. That gap is a sum:

• The endpointing decision, meaning how long the system sits in silence before it accepts the turn is over. Usually the biggest slice, and configured rather than fixed.
• Speech to text, turning the caller's audio into words.
• The model, working out what to say.
• Text to speech, up to the first audible syllable.
• Network transit in both directions, plus the de-jitter buffer that each hop uses to smooth out packet arrival. ITU-T G.114 tells network planners to count half of that buffer's peak delay against their budget.

Three layers sit under that first bullet, and vendors use the words loosely. Soniox's wiki separates them: VAD (voice activity detection) asks whether the sound is speech at all, endpoint detection asks whether the speaker has finished, and turn detection asks whose turn it is.

Only the middle one gets your caller cut off.

Web teams have had this argument for years under another name, which is roughly what a chat widget does to a page. Phones are less forgiving. An agent that takes an extra second doesn't sound slow. It sounds like it wasn't listening.

Why does my AI receptionist pause before it answers?

Because it's waiting out a timer, and the timer only starts once you stop making noise.

LiveKit's agent docs put numbers on it. Their session waits between 0.5 and 3.0 seconds of silence to confirm a turn has ended. With the audio turn detector enabled, that range drops to 0.3 to 2.5 seconds, and both ends can be overridden. Fixed endpointing always uses the configured numbers; a dynamic mode moves inside the range based on the pauses in that call.

So half a second of the pause your caller hears can be spent before the model has been asked anything at all.

The two pause settings that stack

Here is the part that does not come up in demos. Transcription providers do their own endpointing, and the agent framework does it again on top.

AssemblyAI's guide for running its speech model on LiveKit lists three presets. The default, balanced, uses a 128 ms minimum silence, a 1280 ms maximum, and a 500 ms interruption delay. The guide then notes that LiveKit's own default minimum delay of 0.5 seconds applies on top of that, and that setting it to zero avoids the extra wait.

Two components, both waiting politely for the other. Nobody chose a 628 ms floor. It's what you get when two sensible defaults meet.

Why does the agent talk over my callers?

Because the silence timer expired while the caller was still thinking, or because the interrupt detector heard something that wasn't a caller.

Pauses in real speech aren't endings. People stop in the middle of a gate code, halfway through an address, in the middle of spelling a surname, which causes its own trouble further down the line, as we covered in why agents get names wrong. A plain silence timer reads all of those as the end of a turn.

Builders hit it too. One Vapi community thread starts "Struggling With Bot Cutting Off Callers", posted by a developer whose agent still interrupted people giving addresses at the maximum three second pause setting.

Some platforms now split the setting by what is being said. Telnyx's release notes for its Start Speaking Plan describe separate wait times for digit sequences and for pauses in unpunctuated speech, because people read numbers slowly. That's the right shape of fix. An agent should be more patient about "four, one, seven" than about "yeah, that works".

The other cause is the agent hearing itself. AssemblyAI's guide says an agent that interrupts itself is picking up its own speech, and points at the voice activity threshold and the interruption delay. On a phone call that usually means echo in the audio path rather than a bad setting.

How much voice AI latency is acceptable on a phone call?

Under 150 ms one way on the network, and no single correct number for the gap between turns. There are two budgets, so there are two answers.

On the network, the target is old and specific. ITU-T G.114, the recommendation telecoms networks are planned against, puts delays under 150 ms at the point where most applications get what it calls essentially transparent interactivity, and tells planners not to exceed 400 ms one way whatever the application. The current edition is from 2003. The physics of a conversation hasn't moved. All of that is transmission, spent before anything in your agent decides a caller has stopped talking.

On the conversation there is no correct number, and a vendor quoting one without asking what your callers say is guessing. What you can measure is the trade: how often the agent cuts someone off, against how much dead air a patient setting buys you. Count both on your own line for a week.

How do I test my AI phone agent's timing in six calls?

Call your own number six times, from the handsets your callers actually use, with a script in front of you. Twenty minutes, no tooling.

1. Book a normal job with no pauses. Stopwatch from the moment you stop speaking to the first syllable back. Three times, keep the worst.
2. Read out a twelve digit reference number with a deliberate two second pause in the middle. Note whether you got cut off.
3. Give an address the way people really give addresses. "Flat 3, 114, sorry, hang on, Mill Road."
4. Interrupt the agent mid-sentence. It should stop inside about half a second, and keep what you said instead of making you repeat it.
5. Call from a mobile in a van with the window down. Count the times the agent reacts to noise that isn't you.
6. Ask for a person and time the handoff, ringing included. Transfers break in their own ways, which we went through in what happens when a transfer fails.

Then ask for the settings in writing, by parameter name and value: minimum silence, maximum silence, interruption delay, and whether anything upstream adds its own wait. A vendor who can't produce those four numbers isn't tuning your agent. They're hoping.

Where chasing low latency is the wrong goal

Snappy and good are different things, and which one you want depends on your callers' sentences.

A restaurant taking table bookings wants snappy: short utterances, tiny vocabulary, and a long wait after "table for four, Friday, eight" reads as slow-witted. A dental clinic in Dubai Marina wants patience, because callers spell names in a second language, read out Emirates ID digits, and stop mid-sentence to check with whoever is in the room. For those callers a 200 ms setting is a worse agent, even though it wins the side-by-side demo.

Which costs us work. We default to patient settings, so our demo agent sounds slower than the one a prospect heard the week before, and we've lost that comparison more than once.

We also can't tune our way out of a bad line. When an office phone runs over consumer broadband with no traffic priority, the delay and the jitter sit upstream of every parameter we control. The honest answer is to fix the line first, which stalls the project and the invoice with it.

If you take nine calls a day and someone is usually free to pick up, a human beats every setting on this page. Buy an agent when the phone is beating your staff, not before.

What's changing in the next year

Two things are already scheduled.

The pipeline is collapsing. OpenAI made its Realtime API generally available on 28 August 2025 alongside gpt-realtime, a model that handles audio directly instead of chaining speech to text, a language model and text to speech. The same release added calling over SIP (Session Initiation Protocol, the standard carriers use to set up calls). Newer realtime models have followed, and OpenAI has said the older beta interface will eventually be retired, so platforms built on it have to move.

That won't fix interruptions. A single model still has to decide when you've stopped talking, and that decision is still a configured tolerance for silence. The trade-off in this post outlives the architecture change.

For UK businesses, the audio path itself changes. Openreach retires the copper PSTN, the analogue phone network, on 31 January 2027. It says services including ISDN and Featureline, the line products a lot of older office phone systems still sit on, stop working on that date unless they move to IP.

Migrate a line and commission an agent in the same year, and you should measure the timing again afterwards. Check what else is plugged into that line too. Alarms and card terminals have their own switchover.

Make the six calls this week. Write down two numbers: how often the agent cut you off, and the longest gap you heard. Those are the whole argument with whoever built it. We build and tune AI voice agents, so if you'd rather we made the calls, send us the number and we'll come back with the settings worth changing, ours or not.