Automated Chatbot Regression Testing: 20 Golden Questions to Re-Check After Every Change
Automated chatbot answers don’t stay correct on their own. You change a price, add a new delivery partner or update a return rule, and an answer that was right last week quietly becomes wrong. Software teams call the fix regression testing: a fixed set of questions you re-ask after every change. This guide shows a small business how to build its own set of golden questions, run it in ten minutes, and catch problems before customers do.
New festive prices added to the website.
18 of 20 pass. Two old prices still appear.
Old price page removed from the bot’s sources.
No wrong quotes, no awkward refunds.
⚡ In short
- Every change to your prices, policies, pages or bot knowledge can break answers that used to be right.
- Keep 15 to 25 golden questions, each with the facts a correct answer must contain and the phrases it must never contain.
- Re-run the set after every change, and monthly even without changes. It takes about ten minutes.
- Use the free checker on this page to mark answers pass or fail in seconds.

The problem
Why correct answers go wrong without anyone touching the bot
Most people think of an automated chatbot as something you set up once. You add your FAQs, test a few questions, and switch it on. The bot answers well, and you stop thinking about it.
Then the business changes. Your courier raises rates, so delivery is no longer free under ₹499. You stop offering COD on large orders. A doctor at your clinic changes days. You add a new branch. Each change is small, and you update the website. But the bot may still be reading an old page, an old document or an old saved reply, so it keeps giving the old answer, confidently and politely.
There are three common ways this happens:
- Two sources disagree. You updated the pricing page but not the FAQ page, and the bot reads both. Sometimes it picks the old one.
- An old document is still loaded. Last year’s brochure or price list is still in the bot’s knowledge, even though the website changed.
- A new rule has an exception the bot doesn’t know. “Free delivery over ₹999” is clear, but if bulky items are excluded and that’s only written on the product page, the bot may promise free delivery on a sofa.
None of these show up when you test only the new question. They show up in questions you didn’t think to re-check. That’s why a fixed set, re-run every time, works better than ad-hoc testing.
Golden questions
What makes a question golden
Not every question deserves a place in your automated chatbot test set. Pick ones that meet at least one of these.
Asked often
Your top ten most common questions. If one of these breaks, many customers see it within hours.
Expensive if wrong
Prices, discounts, refunds, delivery charges, fees, warranty terms. A wrong answer costs money or trust.
Changes often
Anything seasonal or time-bound: festival offers, holiday timings, exam batches, monsoon closures.
Has an exception
“Free delivery, except bulky items.” “COD, up to ₹5,000.” Exceptions are what bots drop first.
Must be refused
Questions the bot should not answer, such as medical advice or promises about refunds, to check it still hands them over.
Mixes two topics
“Can I pay COD and get it by Friday?” Combined questions show whether the bot handles both parts.
Building your set
Write each golden question with a pass rule
A golden question isn’t just a question. It’s a question plus a clear rule for what counts as a correct answer, so anyone on your team can judge it the same way.
1. Write the question in a customer’s words
Use real phrasing from your chats: “cod hai kya” or “do u deliver to mysore”, not “What payment methods do you accept?” Bots that handle polished questions sometimes stumble on real ones.
2. List the must-contain facts
Write the two or three facts a correct answer must include. For delivery to Pune: “3–5 working days” and “Pune”. For COD: “₹5,000” (the limit). Keep facts short and exact. They’re what you’ll check for.
3. List the must-not-contain phrases
Write any phrase that would make the answer wrong or risky. For sale returns: “full refund”. For delivery: “free delivery everywhere”. This catches the old or invented answers.
4. Note the source
Write down where the correct answer lives: which page, document or saved reply. When the test fails, you’ll know exactly what to fix.
5. Mark questions that should be handed over
For risky topics, the pass rule is the handover itself: the answer must say a person will help, and must not attempt a solution.
Fifteen to twenty-five questions is the right size for a small business. Fewer and you’ll miss things; more and you’ll stop running it.
Free tool
Check your bot’s answers against your rules
Three sample questions are filled in, and two of them fail on purpose. Press Check to see why, then replace them with your own.
For each question: list the facts a correct answer must contain (separate with commas), and the words it must never contain. Paste what your bot actually replied, then press Check. Nothing leaves your browser.
Starter set
20 golden questions you can adapt
Copy the set closest to your business. Replace the facts with your own.
When to run it
Five moments that should trigger a re-run
The value of a golden set comes from running it at the right moments. Put these in your routine:
After any change to prices, fees or offers
This is the most common cause of wrong answers. Run the full set, not just the price questions, because offers often change delivery and return answers too.
After editing the bot’s knowledge
Adding a new page or document can change how the bot answers older questions, especially if the new content mentions the same topics in different words.
After a website update
A redesign, a new theme or reorganised pages can change what the bot reads. Re-run before and after if you can.
Before a busy season
Run it a week before Diwali, a sale, admissions season or any peak. Wrong answers during a rush multiply quickly; our surge playbook for peak days covers the rest of the preparation.
Once a month regardless
Even if nothing seemed to change, run it. Somebody may have edited a page, or an old document may resurface. A monthly run takes ten minutes and catches drift early. Add it to the calendar alongside the other checks in the maintenance calendar.
Ten-minute routine
How to run the set
The same steps every time, so the results are comparable.
1
Open a test chat
Use your own website in a private browser window, as a visitor would see it, not a preview inside the dashboard.
2
Ask each question
Copy the question exactly as written in your set. Start a fresh chat for each one so earlier answers don’t influence later ones.
3
Paste the answer
Copy each reply into the checker on this page or your spreadsheet.
4
Mark pass or fail
Use the must-contain and must-not-contain rules, not your gut feeling.
5
Fix at the source
For each failure, correct the page, document or saved reply the answer came from. Remove outdated sources.
6
Re-run failures
Ask the failed questions again. Only call it done when they pass.
Fixing failures
How to find what caused a wrong answer
When a golden question fails, resist the temptation to add a special rule or a new FAQ entry that patches over it. Patches pile up and contradict each other. Find the real cause instead.
Look for the old fact
Search your website, documents and saved replies for the wrong fact the bot gave, such as the old price or the old timing. Wherever it appears, update or remove it. In most cases, the bot is faithfully repeating something you still have published somewhere.
Look for two versions of the truth
If the correct fact and the wrong fact both exist, the bot may switch between them. Keep one source per topic, as explained in our guide to writing knowledge your bot can use.
Look for the missing exception
If the bot gave the general rule but missed the exception, add the exception next to the rule in the same paragraph. “Free delivery over ₹999 (except furniture, which is ₹499 flat)” works better than putting the exception on a different page.
Check the handover list
If a question that should be handed over got answered, check your handover keywords and flows. See conversation bot handover for patterns that work.
Record the fix
Write down what failed, why, and what you changed. After a few months, this log shows you which parts of your business information are most fragile.
Keep a record
A simple results log
One row per run. The trend matters more than any single result.
| Date | Trigger | Passed | Failed questions | Cause | Fix |
|---|---|---|---|---|---|
| 2 Oct | Monthly run | 19 / 20 | Sale returns | Old FAQ said full refund | FAQ updated |
| 14 Oct | Diwali prices | 17 / 20 | Delivery charge, 2 prices | Old price PDF still loaded | PDF removed |
| 15 Oct | Re-run | 20 / 20 | — | — | — |
| 1 Nov | Monthly run | 20 / 20 | — | — | — |
Illustrative log for a made-up shop.
Beyond the set
Signals that your golden set needs new questions
Your golden set should grow and change with your business. Add a question when:
- A customer complains about a wrong answer. Every real mistake becomes a permanent test, so it can’t happen again.
- You add a product, service, branch or policy. Add one or two questions about it straight away.
- A topic suddenly appears often in chats. Monthly chat reviews, like the routine in our guide on AI driven chatbots as a data source, show you new frequent topics.
- A team member corrects the bot in a handover. If your team often says “actually, that’s not right”, that topic needs a golden question.
And remove a question when the topic no longer exists, so the set stays short enough to run.
An automated chatbot is only as reliable as the checks behind it. Golden questions turn “I hope it’s still right” into “I checked on Monday, and it’s right.” That’s the confidence you need to leave the bot answering at night and on holidays.
In LiveChatBot
Running golden questions on your own widget
In LiveChatBot, the AI Agent answers from the knowledge you give it, and the free keyword bot answers from your saved replies when AI replies aren’t available. Test both: ask your golden questions once with AI replies on, and check that your most important saved replies are up to date for the fallback.
Open your website in a private window to see exactly what visitors see. Every test chat appears in your inbox like a real conversation, so you can review the full answer later. Visitor tracking shows the page the test came from, which helps when an answer depends on the page.
The free plan covers two websites with 100 visitors per website each month, the AI Agent with a small monthly allowance of AI replies, and the free keyword bot. Start on livechatbot.in, build your knowledge with how to train your AI chatbot, then write your first golden set before you announce the chat to customers.
Frequently asked questions
What is regression testing for an automated chatbot?
Re-asking a fixed set of important questions after every change, to make sure answers that were correct before are still correct.
How many golden questions does an automated chatbot need?
Fifteen to twenty-five for most small businesses: your most common questions, the expensive ones, the seasonal ones and the ones the bot must hand over.
How often should I run the set?
After every change to prices, policies, pages or knowledge, before busy seasons, and once a month regardless.
Do test chats count as visitors?
They can, because a test chat comes from a browser just like a real visitor. Run your whole set in one sitting from one private window, and it uses very little of your monthly visitor allowance.
What if the bot passes but customers still get wrong answers?
Add the real customer question, in their exact words, to your golden set. Real phrasing often reveals gaps that polished test questions miss.
Keep reading
Related guides
Auto Chat Bot Audit: Sort Your Chats Into Automate, Assist and Never-Automate in One Afternoon
Artificial Intelligence Chatbot Project: A Practical Blueprint from Brief to Demo
Application Chatbot for Admissions, Jobs and Loans: Help People Finish the Form Instead of Abandoning It
Catch wrong answers before your customers do
Write twenty golden questions, run them after every change, and keep your bot trustworthy. Start free on livechatbot.in.