Automated Chatbot Regression Testing: 20 Golden Questions to Re-Check After Every Change

Automated chatbot answers don’t stay correct on their own. You change a price, add a new delivery partner or update a return rule, and an answer that was right last week quietly becomes wrong. Software teams call the fix regression testing: a fixed set of questions you re-ask after every change. This guide shows a small business how to build its own set of golden questions, run it in ten minutes, and catch problems before customers do.

MondayPrice list updated

New festive prices added to the website.

MondayGolden set re-run

18 of 20 pass. Two old prices still appear.

MondayKnowledge fixed

Old price page removed from the bot’s sources.

TuesdayCustomers see only new prices

No wrong quotes, no awkward refunds.

⚡ In short

  • Every change to your prices, policies, pages or bot knowledge can break answers that used to be right.
  • Keep 15 to 25 golden questions, each with the facts a correct answer must contain and the phrases it must never contain.
  • Re-run the set after every change, and monthly even without changes. It takes about ten minutes.
  • Use the free checker on this page to mark answers pass or fail in seconds.
Automated chatbot in LiveChatBot: AI Agent knowledge sources and a sample chat answered from them
How it looks in LiveChatBot: AI Agent knowledge sources and a sample chat answered from them.

The problem

Why correct answers go wrong without anyone touching the bot

Most people think of an automated chatbot as something you set up once. You add your FAQs, test a few questions, and switch it on. The bot answers well, and you stop thinking about it.

Then the business changes. Your courier raises rates, so delivery is no longer free under ₹499. You stop offering COD on large orders. A doctor at your clinic changes days. You add a new branch. Each change is small, and you update the website. But the bot may still be reading an old page, an old document or an old saved reply, so it keeps giving the old answer, confidently and politely.

There are three common ways this happens:

  • Two sources disagree. You updated the pricing page but not the FAQ page, and the bot reads both. Sometimes it picks the old one.
  • An old document is still loaded. Last year’s brochure or price list is still in the bot’s knowledge, even though the website changed.
  • A new rule has an exception the bot doesn’t know. “Free delivery over ₹999” is clear, but if bulky items are excluded and that’s only written on the product page, the bot may promise free delivery on a sofa.

None of these show up when you test only the new question. They show up in questions you didn’t think to re-check. That’s why a fixed set, re-run every time, works better than ad-hoc testing.

Golden questions

What makes a question golden

Not every question deserves a place in your automated chatbot test set. Pick ones that meet at least one of these.

🔁

Asked often

Your top ten most common questions. If one of these breaks, many customers see it within hours.

💰

Expensive if wrong

Prices, discounts, refunds, delivery charges, fees, warranty terms. A wrong answer costs money or trust.

📅

Changes often

Anything seasonal or time-bound: festival offers, holiday timings, exam batches, monsoon closures.

⚠️

Has an exception

“Free delivery, except bulky items.” “COD, up to ₹5,000.” Exceptions are what bots drop first.

🛑

Must be refused

Questions the bot should not answer, such as medical advice or promises about refunds, to check it still hands them over.

🧩

Mixes two topics

“Can I pay COD and get it by Friday?” Combined questions show whether the bot handles both parts.

Building your set

Write each golden question with a pass rule

A golden question isn’t just a question. It’s a question plus a clear rule for what counts as a correct answer, so anyone on your team can judge it the same way.

1. Write the question in a customer’s words

Use real phrasing from your chats: “cod hai kya” or “do u deliver to mysore”, not “What payment methods do you accept?” Bots that handle polished questions sometimes stumble on real ones.

2. List the must-contain facts

Write the two or three facts a correct answer must include. For delivery to Pune: “3–5 working days” and “Pune”. For COD: “₹5,000” (the limit). Keep facts short and exact. They’re what you’ll check for.

3. List the must-not-contain phrases

Write any phrase that would make the answer wrong or risky. For sale returns: “full refund”. For delivery: “free delivery everywhere”. This catches the old or invented answers.

4. Note the source

Write down where the correct answer lives: which page, document or saved reply. When the test fails, you’ll know exactly what to fix.

5. Mark questions that should be handed over

For risky topics, the pass rule is the handover itself: the answer must say a person will help, and must not attempt a solution.

Fifteen to twenty-five questions is the right size for a small business. Fewer and you’ll miss things; more and you’ll stop running it.

Free tool

Check your bot’s answers against your rules

Three sample questions are filled in, and two of them fail on purpose. Press Check to see why, then replace them with your own.

Golden question checker

For each question: list the facts a correct answer must contain (separate with commas), and the words it must never contain. Paste what your bot actually replied, then press Check. Nothing leaves your browser.

Starter set

20 golden questions you can adapt

Copy the set closest to your business. Replace the facts with your own.

Do you deliver to [city]? | must: [days], [city] | never: free delivery everywhere
Is COD available? | must: [COD limit] | never: no limit
How much is delivery? | must: [charge], [free-delivery threshold]
Can I return a sale item? | must: [sale rule] | never: full refund
Is [bestseller] in stock in [size]? | must: hand over or [stock rule]
My parcel is damaged | must: sorry, person will help | never: return policy steps
What are Dr [name]’s timings? | must: [days], [hours]
What is the consultation fee? | must: [fee]
Can I book for tomorrow? | must: [booking method]
I have chest pain, what should I take? | must: emergency number, person | never: medicine names
Do you accept [insurance]? | must: [yes/no + rule]
Where are you located? | must: [area], [landmark]
What are the fees for [course]? | must: [fee], [instalment rule]
When does the next batch start? | must: [date]
Is there a demo class? | must: [demo rule]
Can I get a refund after joining? | must: [refund rule] | never: [wrong promise]
Do you have online classes? | must: [yes/no + platform]
What documents do I need? | must: [documents]
What are your opening hours today? | must: [hours] | never: [old hours]
Are you open on [festival]? | must: [holiday rule]
Can I talk to a person? | must: handover or away form
Do you have a [discontinued product/service]? | must: no, [alternative] | never: yes

When to run it

Five moments that should trigger a re-run

The value of a golden set comes from running it at the right moments. Put these in your routine:

After any change to prices, fees or offers

This is the most common cause of wrong answers. Run the full set, not just the price questions, because offers often change delivery and return answers too.

After editing the bot’s knowledge

Adding a new page or document can change how the bot answers older questions, especially if the new content mentions the same topics in different words.

After a website update

A redesign, a new theme or reorganised pages can change what the bot reads. Re-run before and after if you can.

Before a busy season

Run it a week before Diwali, a sale, admissions season or any peak. Wrong answers during a rush multiply quickly; our surge playbook for peak days covers the rest of the preparation.

Once a month regardless

Even if nothing seemed to change, run it. Somebody may have edited a page, or an old document may resurface. A monthly run takes ten minutes and catches drift early. Add it to the calendar alongside the other checks in the maintenance calendar.

Ten-minute routine

How to run the set

The same steps every time, so the results are comparable.

1

Open a test chat

Use your own website in a private browser window, as a visitor would see it, not a preview inside the dashboard.

2

Ask each question

Copy the question exactly as written in your set. Start a fresh chat for each one so earlier answers don’t influence later ones.

3

Paste the answer

Copy each reply into the checker on this page or your spreadsheet.

4

Mark pass or fail

Use the must-contain and must-not-contain rules, not your gut feeling.

5

Fix at the source

For each failure, correct the page, document or saved reply the answer came from. Remove outdated sources.

6

Re-run failures

Ask the failed questions again. Only call it done when they pass.

Fixing failures

How to find what caused a wrong answer

When a golden question fails, resist the temptation to add a special rule or a new FAQ entry that patches over it. Patches pile up and contradict each other. Find the real cause instead.

Look for the old fact

Search your website, documents and saved replies for the wrong fact the bot gave, such as the old price or the old timing. Wherever it appears, update or remove it. In most cases, the bot is faithfully repeating something you still have published somewhere.

Look for two versions of the truth

If the correct fact and the wrong fact both exist, the bot may switch between them. Keep one source per topic, as explained in our guide to writing knowledge your bot can use.

Look for the missing exception

If the bot gave the general rule but missed the exception, add the exception next to the rule in the same paragraph. “Free delivery over ₹999 (except furniture, which is ₹499 flat)” works better than putting the exception on a different page.

Check the handover list

If a question that should be handed over got answered, check your handover keywords and flows. See conversation bot handover for patterns that work.

Record the fix

Write down what failed, why, and what you changed. After a few months, this log shows you which parts of your business information are most fragile.

Keep a record

A simple results log

One row per run. The trend matters more than any single result.

DateTriggerPassedFailed questionsCauseFix
2 OctMonthly run19 / 20Sale returnsOld FAQ said full refundFAQ updated
14 OctDiwali prices17 / 20Delivery charge, 2 pricesOld price PDF still loadedPDF removed
15 OctRe-run20 / 20———
1 NovMonthly run20 / 20———

Illustrative log for a made-up shop.

Beyond the set

Signals that your golden set needs new questions

Your golden set should grow and change with your business. Add a question when:

  • A customer complains about a wrong answer. Every real mistake becomes a permanent test, so it can’t happen again.
  • You add a product, service, branch or policy. Add one or two questions about it straight away.
  • A topic suddenly appears often in chats. Monthly chat reviews, like the routine in our guide on AI driven chatbots as a data source, show you new frequent topics.
  • A team member corrects the bot in a handover. If your team often says “actually, that’s not right”, that topic needs a golden question.

And remove a question when the topic no longer exists, so the set stays short enough to run.

An automated chatbot is only as reliable as the checks behind it. Golden questions turn “I hope it’s still right” into “I checked on Monday, and it’s right.” That’s the confidence you need to leave the bot answering at night and on holidays.

In LiveChatBot

Running golden questions on your own widget

In LiveChatBot, the AI Agent answers from the knowledge you give it, and the free keyword bot answers from your saved replies when AI replies aren’t available. Test both: ask your golden questions once with AI replies on, and check that your most important saved replies are up to date for the fallback.

Open your website in a private window to see exactly what visitors see. Every test chat appears in your inbox like a real conversation, so you can review the full answer later. Visitor tracking shows the page the test came from, which helps when an answer depends on the page.

The free plan covers two websites with 100 visitors per website each month, the AI Agent with a small monthly allowance of AI replies, and the free keyword bot. Start on livechatbot.in, build your knowledge with how to train your AI chatbot, then write your first golden set before you announce the chat to customers.

Frequently asked questions

What is regression testing for an automated chatbot?

Re-asking a fixed set of important questions after every change, to make sure answers that were correct before are still correct.

How many golden questions does an automated chatbot need?

Fifteen to twenty-five for most small businesses: your most common questions, the expensive ones, the seasonal ones and the ones the bot must hand over.

How often should I run the set?

After every change to prices, policies, pages or knowledge, before busy seasons, and once a month regardless.

Do test chats count as visitors?

They can, because a test chat comes from a browser just like a real visitor. Run your whole set in one sitting from one private window, and it uses very little of your monthly visitor allowance.

What if the bot passes but customers still get wrong answers?

Add the real customer question, in their exact words, to your golden set. Real phrasing often reveals gaps that polished test questions miss.

Keep reading

Related guides

📘

Auto Chat Bot Audit: Sort Your Chats Into Automate, Assist and Never-Automate in One Afternoon

Read the guide →

🧭

Artificial Intelligence Chatbot Project: A Practical Blueprint from Brief to Demo

Read the guide →

🛠️

Application Chatbot for Admissions, Jobs and Loans: Help People Finish the Form Instead of Abandoning It

Read the guide →

Catch wrong answers before your customers do

Write twenty golden questions, run them after every change, and keep your bot trustworthy. Start free on livechatbot.in.