A client came to us with 400 support tickets a day and a two-person team drowning in them. The ask was simple: route tickets to the right queue automatically. We had a working classifier in about a week. Making it something the support team would actually rely on took another three months.
The model was the easy part
A prompt with fifteen labelled examples got us to 94% accuracy on a held-out set. That number felt like the finish line. It wasn't. The 6% it got wrong were not evenly distributed — they clustered almost entirely in billing disputes and account-access issues, which happen to be the two categories where a misroute costs the most.
Aggregate accuracy hid the only thing that mattered. We started measuring per-category recall instead, and the picture changed completely: 98% on general enquiries, 71% on billing.
What we changed
Three things moved the needle, and none of them were prompt tweaks.
We let it abstain. Any classification below a confidence threshold goes to a human queue instead of guessing. That dropped automated coverage from 100% to 82% and raised trust enormously, because the failures the team saw were now honest ones.
We wrote an evaluation set before we wrote the prompt. Two hundred real tickets, hand-labelled by the support lead. Every prompt change ran against it. Without that, "this feels better" was the only available metric, and it is worthless.
We logged the reasoning. When a ticket is misrouted, the team can see why the model chose what it chose. That turned an opaque system into one people could argue with — and arguing with it is how the label definitions got fixed.
What we'd tell you before you start
Budget most of your time for evaluation, feedback loops and failure handling. The demo takes a week. The system takes a quarter. If a vendor tells you otherwise, ask them what their per-category recall looks like on the categories that cost money.
- #LLM
- #Support
- #Automation
- #Evaluation