Rebuilding our booking chatbot around a decision model

We run the chatbot that books service appointments for one of Zagreb’s automobile clubs, over web chat and WhatsApp. In September we rewrote it, moving it from Botpress’s visual editor into code.

In the old version, about half of all messages went through an LLM call, and that was before counting the OpenAI Assistant that took over whenever a customer asked for a different time. In the new one it’s about one in ten. Customers get the same kind of conversation as before. The difference is that most messages are now handled by a decision model called Jev instead of an LLM.

Most messages aren’t open questions

A booking conversation is mostly the bot asking something with a handful of sensible answers. Which service do you need? Are you a club member? Does one of these times work? Someone who answers “yes, of course” to the membership question doesn’t need a model that could write them an essay. They need someone to notice that they said yes.

That’s what decision models do. You give one some context and a question with a fixed set of possible answers, and it returns a probability for each answer. It never writes a sentence, so there’s nothing to parse and nothing for it to make up. Jev answers our questions in about a quarter of a second, at least five times faster than an LLM call in the same bot, and only the input is billed.

Models like this are fairly new, but they’re spreading quickly and people keep finding new uses for them. Jev, from TypeSafe, is one of the first, and OpenAI recently introduced its own. They aren’t a replacement for an LLM. Jev can’t write, summarise or explain its reasoning, and it takes at most 64,000 tokens of context. It’s meant to make the quick calls in front of a bigger model, which is exactly the job we had for it.

The model reads, the code decides

At every step, the bot sends Jev the question it just asked and the answers that make sense there, plus a few that always do: start over, change something chosen earlier, a question about something else, and none of these.

What happens next is up to the code. A clear winner is acted on. If two answers are close, the bot asks which one the customer meant: “I was a member last year” to “Are you a member?” gets a question back rather than a guess. Anything less clear, and the bot asks again. Jev never changes anything by itself. The booking, the availability, the SMS check and the final confirmation all stay in code.

Turning “next Thursday” into a date

Time was the part we were least sure about. “Next Thursday, around half nine?” looks like a job for an LLM, or at least for a date parser.

It turned out Jev handles it well, as long as we ask the right questions. It copes with a long list of possible answers if each one is clearly described, so the answers can be the calendar itself: the coming weeks, and the days in them, each with a short note on what it is. And one question can depend on the answer to another. We first ask which week the customer means, and only then which day of it and what time. Each question is small enough to answer reliably, and together they arrive at Thursday 22 October, 9:30.

Jev never does the date arithmetic. It picks from dates the code has already worked out, and the code runs the search. That was the main thing we took from this part: a decision model can work out a lot more than yes or no, if the questions are put together well.

When a list isn’t enough

Some requests don’t fit any list. “Any day except Monday” or “not before 10 and not after 12” have no answer among weeks, days and times. So Jev answers one more question: can these answers hold everything the customer said? When they can’t, the message goes to an LLM, which turns it into a search, and the code checks every value before using it.

An LLM also writes the reply whenever a customer types something, so “is 1pm free as well?” gets a straight yes or no rather than a stock list of times. That’s why the conversation still feels the same. The LLM is still there to talk; it just no longer decides what happens.

What changed

Before the switch, 51% of the old bot’s messages came with an LLM call, not counting the Assistant. In the new bot it’s 11%. Per message from a customer, that’s from more than one LLM call to about one in four.

That means far fewer moments where a customer waits for a language model, and with the same model it would be about a fifth of the bill. In the first four days, about half of all conversations ended with a booking in the calendar.

This works for a conversation with clear steps, where every step has a right answer and the last one writes into a real calendar. A bot meant to chat about anything, say an assistant answering open questions about a product, has no list of answers that could cover the conversation. There an LLM would lead, and a decision model would only help with smaller calls along the way, like where to route a question.

Chatbots are only one place for a model like this. People are already using them to route requests, rank search results, check citations against their sources or even play Subway Surfers. Ours is a small, practical one: most of what customers type in a booking chat never needed an LLM, and the bot is faster without it.