At LendEasy.ai, we got access to Jev this past Friday, and over the weekend we rewrote our key classifiers in Jev and improved our cost, our latency and also our eval results significantly. All of this while simplifying our code and improving our design.
Jev is a truly groundbreaking zero-shot classifier. If you haven’t read about it yet, it is time to catch up to the hype because it is going to change how people use LLMs in code. It makes classification dirt cheap (up to 15x cheaper as compared to LLMs on cached context), crazy fast (up to 6x faster, as we observed), structured by design, and without hallucination (it cannot invent a category, because it only picks from the options you give it). Here, we will go over where we placed Jev into our existing architecture.
We are building a voice agent for doing collections. This is a regulated field and we need to take customers through regulated pre-defined workflows where they can decide to pay, postpone payments, declare hardship or dispute payment details. A typical conversational flow, requires INTENT classification. In a regulated app like ours, intent classification alone is not enough: we also need to determine if a caller has changed their mind. For example, A caller may decide to pay the partial monthly amount, then realize that it may impact their credit score and decide to pay the entire amount the same day. We call these as PIVOTs. We want to classify when a pivot occurs, so we can wipe the slate clean of any pending orders. We also model a particular series of questions given to our borrowers as a LADDER, where each step is a choice in a multi-turn conversation that a caller may have to make. Together, we need to analyze, the INTENT, a PIVOT (change in direction), and where we are in a LADDER (which step in a single conversation).
For example, a borrower may declare hardship by saying, “I have lost my job”. They have pivoted from a negotiation to a hardship. They need to be taken down a hardship ladder, where each step may give them a choice where they may be asked to defer the payment by one month, split their payment into multiple installments, or choose to talk to a specialist if none of those options work for them.
Our existing system had two large LLM calls ( >15K tokens) for this. In each we will supply the context of where we were in the conversation, explained each choice in detail, asked the LLM to pick one or more outcomes, and fill over 60 fields including booleans and data fields (like dollar amounts and dates). These LLM calls would need to be done in parallel and took more than one second to resolve even on the fastest inference providers. Most of this could be attributed to the prompt context which we kept enriching all the time for edge cases, and gotchas.
On Thursday last week we were at a conference in Salt Lake City when we started to hear all the buzz around Jev and we applied for access - explaining our use-case to the Jev team. By Friday afternoon we had access to it.
We knew where we wanted to attack the problem: at the classification boundary. Before, even starting we already had a solid set of 337 evals and 84 ladders (multi turn conversations with pivots) - with hardship and payoff as known problem areas under the LLMs. This was going to be used to prove how well we are performing with Jev.
We changed our code incrementally, and with each change we saw our performance get better than our LLM implementation.
We supply Jev with where we are on the call: whether a question is open, what data needs to be collected, what has already been answered, if hardship has been declared and where we are on the hardship ladder (offered step, declined step, and next step), if an existing payment order has been placed and so on. We also clearly state the deal on the table and the last offered deal that has been rejected. We also provide the history of the conversation. We included a set of definitions, which describe in detail what each field means, to help improve the scores.
Then we ask the questions, each question’s options are the outcomes it can choose from. In real conversation, a borrower might express more than one intent that needs to be addressed. Solving this meant we had to work around the key issue with Jev, that it does not do multi-intent classification. This turned out to be fairly straightforward and barely an inconvenience. Jev supports multiple questions in one request, which can be any combination of choice, noul (yes/no), or scoring. So we put each set of mutually exclusive intents into one choice question, with a “none” option when none of the given choices applies.
We gave Jev a total of 28 questions about the current conversation to answer. These questions were about the scope of the response, the shape of money (full, partial, split), timing (today, date, weekday), channel (card, schedule or link), and so on. In a second smaller call, we ask for the intents which help us decide if the regular call flow needs to be interrupted to answer an FAQ, continue the conversation, pivot to a different ladder (from negotiation to hardship), or to bail out due to a compliance risk like an abusive caller or a self-harm risk.
Our results, were simply amazing, we could see that we were now passing 306 of our 337 evals (as compared to 280 with LLMs), and 79 of our 84 ladder cases were passing (as compared to 77 with LLMs). Our most difficult buckets were hardship and payoff evals. For each of these we saw up to 20% improvement in our pass rate - closing the gap significantly. In the cleanest head-to-head comparison, on our corpus of 293 single-label turns (labeled as a golden set) Jev matched the gold label 97.9% of the time against 77.5% for the LLM.
We saw our latency drop from >1 sec to 150 ms for both prompts. Our overall speed of iteration has improved, our code base is simpler, our design more deterministic, and the flakiness (some cases would flip flop a lot on LLMs) disappeared. We still need an LLM call to extract data - but that has gone down from >1 sec of latency to 500 ms - which drastically improves our end user experience over voice. We are also observing a huge cost decrease by 15x already as we still have some LLM calls in our implementation (400x is advertised in per-token price as compared to frontier LLMs).
We feel that this capability is a huge unlock for enterprises and regulated environments. We have tons of ideas on how to use this now for doing things such as sentiment signals at each turn (likelihood to pay, level of frustration), stronger guardrails (as rules can be expressed better and behavior can be flagged for cheap using predefined rules) and one-shot classification throughout our LendEasy.ai stack. Watch this space to see how we are building the future of automation for regulated industries.