Cutting hallucination-driven escalations for a fintech support assistant
The challenge
The team had shipped an LLM assistant to answer customer questions about fees, limits, and account rules. It was fast and popular, but it occasionally stated policy details that were wrong. Each wrong answer created a support escalation and, in a regulated product, a compliance concern.
Our approach
- Mapped the failure surface and ranked answer categories by how costly a wrong response would be.
- Built a golden dataset from real, anonymized support transcripts, focused on fee and policy edge cases.
- Added adversarial prompts that tried to push the assistant into confident guesses.
- Measured hallucination rate and groundedness against the source policy documents.
- Wired the evaluation into CI so a regression blocked the release, and set up drift monitoring after model updates.
Support assistants are one of the most common first uses of an LLM, and one of the easiest to ship without a safety net. This team had done the hard part of getting adoption. The risk was that a confident wrong answer about a fee or a limit could reach a customer, or a regulator.
What we did
We started by writing down the answer categories where a mistake was expensive, then built a golden dataset from real support transcripts so the evaluation reflected how customers actually ask. We added adversarial prompts designed to tempt the model into guessing, and we scored both hallucination rate and groundedness against the source policy documents.
The evaluation runs in continuous integration, so a change that regresses the score cannot ship. After launch, drift monitoring watches for quality changes when the underlying model updates.
Result
The assistant reached release with a hallucination rate under 1 percent on the policy golden set, and the team kept every test suite and dataset in their own repository. The point was not a one-time cleanup. It was a repeatable gate that keeps the assistant honest as it changes.
Representative engagement. Client details are anonymized and figures are illustrative ranges shown to convey the type and breadth of work, not a specific named result.
