Building a voice agent: Fixing the latency problem

The Problem

We define a voice agent as an LLM-driven workflow that sets out to accomplish a particular task for the caller or an organization. Let's take the case of a voice agent making a first-party collections call. Before it can discuss anything about the account, it needs to verify the caller's identity — asking for the ZIP code on file, and falling back to another identifier like date of birth if the caller doesn't have it — deliver the required disclosures, state the past-due balance, answer questions like the payoff amount, and then actually collect a payment or send a secure payment link.

This is a multi-part question and answer session with multiple failure points.

Challenges

There are several challenges here that are unique to an application that is handling a live collections call. Here are the most critical ones:

1. Have a low latency response to any utterance. Typical voice applications need to respond within 1 second — ideally within 700–800 ms of the caller finishing speaking.
2. Callers can ask questions out of order — just as they would with a human. Therefore we can try to fix the logic and workflow — but we cannot be too rigid.
3. Handling barge-in — callers talk over the agent mid-sentence, just as they would with a human. The agent has to stop speaking, listen, and recover without losing its place in the conversation.
4. Negotiating with the caller on how and when they can pay or giving them various options requires the agent to be robust, context aware.
5. Making progress — perhaps the most difficult of them all: the agent must keep making progress towards its goal.
6. No human in the loop — There is no human in the loop who can interrupt and stop the agent from making a wrong decision.

Each of these challenges on its own can derail the conversation and reduce the effectiveness of the call or make the human in the call frustrated enough to end the call.

Low Stakes vs High Stakes Situation

Contrast this with a servicing copilot that supports a loan officer working a case on the servicing plane — drafting messages, summarizing the account, suggesting the next action. That is a low-stakes scenario: the loan officer reviews every suggestion and can step in before anything reaches the borrower. On a live call, the voice agent is in a high-stakes scenario — if the voice agent cannot serve the client properly, the business may end up losing that customer. It can end up as a bad customer review on Google. Worst case, it could lead to a compliance violation for a business in the financial domain.

Sources of Latency

We will first and foremost focus on latency as that is table stakes to get the user interaction acceptable. There are three primary sources of latency here: the speech-to-text system (STT), LLM calls, and text-to-speech (TTS).

The STT system detects the speech and converts it to text. This takes time, adding to the delay before we can even invoke the business logic. This system has to be robust enough to handle intermittent pauses.

A voice agent call can take up to 3-4 different LLM calls. In our setup we use a decider agent that is responsible for invoking one or more LLMs based on conversation history, available agents and a set of hard rules. This is followed by the various LLM calls to formulate a response and other intermediate calls to decide the next activity. While this may also include external tool calling — the majority of the time is spent in LLM calls.

The last step is the TTS system. The TTS system uses the LLM text and sends it to speech conversion and adds to that time taken.

Fixing the Latency problem

We trigger the STT system in streaming mode to reduce the latency of the utterance to text conversion. In this mode the STT streams back the tokens as the caller speaks. Each utterance also comes back with an end-of-turn confidence score, and we can specify an end-of-turn threshold. When the streaming packets come in we work with the embedded end-of-turn signal and use that as further evidence to decide when to end the turn — with a custom waiting period.

This makes sure that we can make a dynamic decision on when to end the turn as pauses are detected — this saves valuable time and also gives us more configurability.

Next is the LLM — the most critical latency bottleneck. We decided to use Cerebras to get the maximum throughput for our latency needs. Cerebras uses wafer scale chips and claims throughput to be in 1000s of tokens per second — much higher than 10s of tokens per second from typical LLM providers. This enables us to get a quick response for our needs. We achieve very low latencies by combining this with careful prompt design: a large static system prompt (which Cerebras caches automatically) and everything else in the user prompt. We use an instruct model with Cerebras, and we leave enough headroom for the prompt, any server-side thinking (which consumes additional tokens), and the specific JSON output we expect from the LLM. On any error, we fall back to a predefined, hardcoded plan to make sure that the user still receives a response and is not left hanging.

The last part is TTS — this is interesting because it leaves no room for streaming the response from the TTS provider. We developed a custom solution to stream the voice output chunk by chunk over our websocket connection. We had to fine tune here to make sure artifacts do not appear if the PCM audio chunks are too small — after a bit of fine-tuning we got latencies low and removed the chunking artifacts that made the audio sound undersampled.

Finally, we account for unforeseen delays when we are fetching data from external services — these can create an awkward silence for the caller. We introduced filler utterances the voice agent can use to keep the caller engaged and the connection alive.