Voice AI
What it takes to build a voice AI assistant that feels natural
Real-time voice AI is mostly an engineering problem of timing. Here is how streaming, turn-taking and interruption handling make an assistant feel like a conversation.
By Manish Lakra · · 7 min read
People forgive a voice assistant for not knowing something. They do not forgive it for awkward silences, talking over them, or ignoring them when they interrupt. Those are timing problems, and timing is where most of the engineering effort in voice AI goes.
The pipeline behind every reply
A voice assistant chains three systems together for every turn of the conversation:
- Speech-to-text turns the caller's words into text.
- A language model, often with access to your data and tools, decides what to say or do.
- Text-to-speech turns the reply back into a voice.
If each step waits for the previous one to finish completely, the delays add up and the caller hears a long pause after every sentence. The fix is streaming at every stage.
Stream everything
Speech recognition should run while the person is still talking, producing partial transcripts as they go. The language model should start generating as soon as the person stops. And text-to-speech should start speaking the first words of the reply while the rest is still being written.
Done well, the caller hears the start of an answer shortly after finishing their sentence, which is close to how people talk to each other.
Knowing when the person has finished
The hardest timing question is deciding when the caller has actually finished speaking. Respond too early and the assistant cuts them off mid-thought. Wait too long and it feels slow.
Simple silence detection is not enough, because people pause to think. Better systems combine the length of the pause with what has been said so far: a sentence that sounds complete can be answered sooner than one that trails off.
Letting people interrupt
In a real conversation, people interrupt. They say "no, the other one" or "actually, wait". An assistant that keeps talking over them feels robotic.
Interruption handling means the assistant keeps listening while it speaks. When the caller starts talking, playback stops, the unspoken part of the reply is dropped, and the system listens to the new input with the conversation so far in mind.
Adding a face
On kiosks and screens, a 3D avatar can make the conversation feel more personal. It adds one more timing requirement: the avatar's lips and expressions have to move in step with the audio. Any drift between the two is immediately noticeable, so the animation has to be driven by the same audio stream the caller hears.
Designing for the real environment
- Background noise in restaurants, stores and streets affects recognition, so test where the assistant will actually be used.
- Accents and mixed languages are normal for many customers; test with real speakers, not only with your team.
- Plan what happens when the connection drops or a system is slow: a short spoken explanation and a way to reach a person.
- Always offer a handover to a human, with the conversation so far passed along.
Where to start
Start with one narrow task, such as taking a booking or answering order-status questions, and measure how many calls the assistant completes on its own. Widen the scope only once that number is where you want it.