Every conversation about AI in customer service starts with tone and ends with tone. Does it sound human, is it too formal, does it use the right words.
Tone is the least of it. The failure that actually costs a business is an answer that was invented: a return window that does not exist, an availability that was never checked, a price from a catalogue the model never saw. It arrives in fluent, confident prose, and the customer has no way to tell it apart from a true one.
Grounding is a property of the pipeline, not a prompt
Telling a model "only use the provided information" helps and does not solve it. What solves it is not giving the model the opportunity.
A grounded answer is assembled: the question is used to retrieve passages from your own material, and the model is asked to answer from those passages and nothing else. When retrieval comes back with nothing relevant, there is no material to answer from, and the correct output is a refusal.
That last step is the one most implementations skip, and it is the one that matters. A system that always produces an answer will always produce one, including for the questions where it should have said it did not know.
What belongs in the material is broader than a FAQ:
- The public pages, because customers ask about what they read there
- Policies, in their current wording, because that is what will be quoted back at you
- Product and price data, from wherever it is authoritative
- Whatever your team already writes by hand five times a week
Answers and actions are different things
There is a line inside every customer conversation, and it is worth drawing explicitly.
Saying is answering a question. The risk is being wrong, and grounding is the control.
Doing is booking, cancelling, refunding, updating a record. The risk is doing the wrong thing to real data, and prose is the wrong mechanism entirely.
Actions should be tool calls with a defined shape: named parameters, validated, executed by code that knows the rules. The model decides that a booking is wanted and with what arguments; it does not write the booking. That separation is what makes it possible to enforce anything at all, because the rules live in the tool and not in a paragraph of instructions the model may or may not follow.
Refusal is a feature, and it needs a destination
"I do not know" is only useful if it goes somewhere. On its own it is a dead end that annoys the customer more than a wrong answer would.
The complete behaviour is: recognise the limit, say so plainly, and hand over with the conversation attached, so the person picking it up does not start by asking what has already been asked. That handoff is the product feature, not the refusal.
The same applies to hesitancy. If the retrieved material is thin or contradictory, answering anyway with a hedge is worse than handing over: a hedged wrong answer is still a wrong answer, and it has now been endorsed.
What to measure
Most teams measure resolution rate, and resolution rate alone rewards exactly the wrong behaviour: an agent that answers everything scores perfectly, whether or not the answers were true.
A usable picture needs at least:
- Resolution rate, but split by whether a human later reopened the conversation.
- Refusal rate. Not zero. A rate of zero means nothing is being caught.
- Handoff quality: how often the human had to re-ask something the customer had already said.
- Flagged answers, reviewed by someone, feeding corrections back into the material rather than into the prompt.
That last distinction matters more than it looks. When an answer is wrong, the fix usually belongs in the knowledge base, because the model answered correctly from material that was incomplete. Fixing it in the prompt hides the gap for one question and leaves it for every other.
Summary
- The expensive failure is confident invention, not clumsy tone.
- Ground the answer in your own material, and let missing material produce a refusal.
- Actions are tool calls with validated parameters. Never prose.
- Refusal is only complete when it hands over with context attached.
- Measure refusals too. An agent that never refuses is not being watched.
