Introduction
One of the most important parts of our technology stack at lendeasy.ai is voice agents. A collections voice agent has to be very carefully designed to make sure that it can ask for payment, negotiate terms with the customer on a smaller amount or on dates — and, most importantly, direct them to an appropriate path in case of dispute or hardship.
A collections voice agent should also be able to handle several kinds of informational questions, FAQs and account data specific questions as well.
In this post I will be covering the thought process of building an agent and the designs we have iterated over the last few weeks of refining our collections voice agent. I will not be covering system-wide reusable pieces like the voice gateway or the workflow engine — those are for another time. This post goes over useful concepts that I have extracted and can be used to build a voice (or even a more general) agent that is driven by data, context and sound data driven decision making principles.
When designing a voice agent we have to handle a lot of uncertainty in what a human (in this case a borrower) might say. A borrower might decide to pay off the entire loan, they may decide to pay only a part of it, they may also change their decision mid call and decide to defer a payment.
A borrower may also ask several informational questions out of order of the conversation line.
The voice agent needs to be able to handle all of these without losing track of where it was, what data it has collected and what next steps it needs to do.
Evals and Requirements
We started with a set of requirements and then consolidated those requirements with evals. These evals helped us ground the most basic request-response expectations from our application that maintain compliance.
A multi-step workflow definition
A typical agentic workflow will have several steps, where each step is a sub-agent, and it will have one or more tool calls for that sub-agent. In the case of a collections workflow, we will have a sub-agent that will verify the identity of the caller, another sub-agent that will do payment negotiation (how much can you pay?), another sub-agent for when the caller discloses hardship, another for authorizing a credit card payment and so on. The workflow edges dictate which node runs next once the current node finishes with a specific code. Some edges pass control back to the user to get an utterance, others will pass control to another subagent which can ask a followup question.
This works if we can figure out which step the borrower needs to be in using a classifier for utterance.
The intent classifier
The intent classifier maps the intent of the borrower’s utterance to one of the sub-agents (steps) of the workflow graph above. The intent classifier tells which specific agent to invoke. This is needed to make sure that partial utterance (like a “Yes”) to an availability question is not taken as a “Yes” for payment authorization. The intent classifier maps the intent explicitly to a step or signals continuation of a step (such as a response, affirmation or denial to a question asked in a previous turn). This also works really well for questions that arrive out of nowhere and are not defined in any workflow sub-agent. These are handled as interrupts.
The interrupt mechanism
The interrupt mechanism exists as a bunch of free-hanging nodes within our workflow graph. These are useful to redirect intent to a node that is normally not in the call flow. This typically maps to a “FAQ” question, which will interrupt the ongoing flow but resume the conversation back to where it was. Another example is, a request to not send any message on the phone, this is an ask that does not fit or change the current conversation flow - but is a preference that can be set out of order and then the regular conversation flow can be resumed.
This acts much like the interrupt mechanism for a processor in the classic computer architecture sense. We stack whatever we are working on - keep it on the side - we service the interrupt and then we resume processing.
A pivot mechanism
So far what we have described works very well when there is a linear flow for a call. However, most of these calls do not go in a linear fashion. A caller, once verified, would like to know what their options are before committing to a particular payment - if at all. Worse, they may pivot completely from one path to another as we have talked above.
These pivots will mean we have to backtrack all the information we have collected so far and start on another thread.
In order to solve this problem, we developed a path classifier. It is similar to the planner in a typical agentic application. The path classifier knows all possible paths that a borrower might take, it then engages the borrower in one of the dedicated paths. The path classifier decides what the call is trying to do. It is different from the workflow edge which tells what next to do, after the current step is done. The current step is decided by the intent classifier. The dedicated paths of the planner can be declared as paths being taken within a sub-agent as well.
In order to test this part thoroughly, we enhanced our test suite with entire conversations that are full of pivots.
Deterministic Pathways
These pathways are deterministic with specific conversation ladders that make it possible to collect information as the conversation keeps going on. These paths are not encoded in a workflow (but are within a sub-agent only). We decided to simplify an agent with a large number of sub-agents to actually a single sub-agent with a large number of paths. We still keep the workflow for where handoffs are real, ex: a sub-agent that is verifying identity is separated from a sub-agent negotiating. The negotiating sub-agent can account for when the caller switches from one path to another. This simplifies the design: we see far fewer bugs from sub-agents passing data to each other, and we maintain fewer per-sub-agent classifiers. We also stop seeing bugs on the seams, since we no longer need to declare every variable and every path created when an utterance forces a switch between sub-agents.
The information that is provided by the borrower is extracted from the conversation by an LLM (same prompt as the planner) and mapped to specific deterministic parameters. In order to help the planner make good judgement we started to give more and more context information about the path we are on to the planner, including what all information we need to collect from the borrower. We capture both explicit and implicit information as variables.
This is the beginning of the data graph for the planner.
The data graph
The data graph feeds the planner with what has happened so far (context memory), what fields are collected, and where we are in the graph. It tells the planner what fields we still need to collect, what we must tell the caller for compliance purposes (the tells), and what obligations, if any, are outstanding.
These are all mapped to specific declarations that are in a json file. For each path, we have a certain set of fields to be collected, and these fields are declared clearly. The corresponding questions for those fields (the asks) are also mapped clearly. The next step is to make sure the agent also tells the caller what they need to hear for compliance purposes (these are the tells). Now we take the memory, the asks, the tells and create a steering context that emits what are the next things to do for the LLM.
Deterministic vs Probabilistic output
We put all of this together and we get an outcome from the planner on where we are right now - namely the path, the fields collected (based on the latest utterance).
Now we have two possible paths, one is to take this and put this all in a deterministic ladder - this ladder keys on the fields from the planner and has specific conditions on what to ask if a particular field is missing based on what fields have been collected within a particular path. This is highly testable and verifiable. We always know what data we need to collect, we know what data is needed for a response from the sub-agent and we know what is the gap - so we can fill the gap deterministically by taking the caller through a response ladder. We have already handled both interrupts and pivots in a declarative way.
The data-graph is a huge asset for us now. We can deterministically traverse it and account for all the cases here.
Similar to the input data graph, we also have a list of deterministic and compliant utterances. These utterances are specific because we need to make sure the borrower hears exactly the amount they have agreed to or they need to know the specific hardship options that they can pick from. There is very little room for hallucinations in this.
We can also combine the input data graph and the output data graph with a separate, dedicated “execution” LLM that picks the utterance from a large set of possible utterances based on the entire input data graph, the steering data, the chosen path, the likely response from the borrower and what fields are already present.
The interesting thing is the deterministic and the probabilistic utterances come very close to each other. We are still analyzing which one to pick and we will keep this blog updated on where we land.
Learnings
The most important learning is that no matter what decision we are taking for execution, we have to capture facts of all the things that are going on in the application. The steering data graph consisting of the asks, the tells and any outstanding obligations keeps the deterministic and probabilistic part of the application on the same page, helps keep the memory updated and helps us verifiably test our outcomes. A simple scan of the required fields against the list of asks lets us find gaps; similarly, at least for deterministic code, we can scan the code to make sure a ladder covers every condition. We also learnt that making pivots, recording them and determining a path via a planner makes our application auditable and easier to debug (we will be talking about debugging a complex application in a later blog post). The key idea is that the state machine is deterministic and encoded; the LLM can’t be relied upon to manage it — we use code for that. The LLM has to be used for classification, routing, extracting fields and (what we are testing now) for utterances. The evals, conversations, workflow definition, intent and planner prompts, all compe together in a configurable way - where only the sub-agents are custom built. We think we have come up with an efficient way to create declarative voice application - that we will elaborate in a future blog post.