We've been called in to rescue enough failed chatbot projects to notice that they fail the same three ways. All three are visible in the sales conversation, months before anything breaks.
Here's what to ask. It works on us too — if we can't answer these clearly for your business, that's useful information about whether to hire us.
1. What happens when it doesn't know?
This is the question that separates a demo from a system. Every agent will eventually meet a question it can't answer: an unusual request, an angry customer, a situation with legal weight. What it does in that moment is most of what your customers will remember.
A good answer is specific. It names the conditions that trigger a handover, says where the conversation goes, and describes what the human sees when it lands. A weak answer talks about accuracy percentages.
If a vendor can't tell you exactly what a customer sees at 2am when the agent gets stuck, they haven't built this before.
What we set up, as a default: the agent hands over when the customer asks for a person, when it detects frustration, when the topic touches money beyond a set threshold, or when it has failed to understand twice. The customer is told plainly that a person will pick this up, and given the hours. The conversation lands in one inbox with the full history attached.
The rule we don't allow anyone to configure away: if a customer asks for a human, they get one.
2. Where does the data go, and who can train on it?
Every conversation your agent has is customer data. Some of it will be health information, financial detail, or the sort of complaint that would be embarrassing in public. You need to know where it sits and who can read it.
Three things to establish before signing:
- Training. Is your data used to improve anyone's model? On business-tier access from the major providers the answer is no by default, but on consumer tiers it often isn't. Ask which tier they're using.
- Retention. How long are transcripts kept, where, and can you delete them on request? "Indefinitely, in a database we manage" is a real answer some vendors give.
- Scope. What can the agent actually read? An agent that has full access to your CRM because it was easier to set up that way is a problem waiting for a bad week.
Ask for it in the contract rather than in an email. A vendor who is comfortable with this will have a DPA ready.
3. Who can change it after launch?
This is the one people forget, and it's the one that quietly kills projects six months in. Your prices change. Your hours change. You add a service and drop another. If every one of those changes requires a support ticket and a two-week wait, the agent starts telling customers things that aren't true, and your team stops trusting it.
The questions worth asking: can someone non-technical on my team change what it says? Is there a written record of what it's been told? If we stop working with you, do I keep the accounts and the configuration, or do they live in your platform?
We write a handover document during the build rather than at the end, and we spend the last week of every project teaching the client's team to make changes themselves. Partly because it's the right thing to do, and mostly because we don't want to be a bottleneck on somebody's Tuesday.
A short version
If you only have ten minutes with a vendor, ask these:
- Show me what a customer sees when the agent gets stuck at 2am.
- Which model tier are you using, and is training disabled on it?
- Who on my team can change what it says, and how long does that take?
Clear answers to those three won't guarantee a good project. But vague answers reliably predict a bad one.
Working through this for your own business? The AI Readiness Checklist covers the same ground in twelve questions, or you can book a free consultation and we'll go through it with you.