Chatbot Python Testing: Score Your Intent Matcher With a Labelled Test Set, pytest and a Confusion Table
Chatbot Python tutorials usually stop when the bot answers its first question. The harder part is knowing whether it answers the next hundred correctly. This guide shows how to build a small labelled test set from real customer questions, score your bot with a short evaluate script, turn the results into pytest tests that fail when a change breaks something, choose a confidence threshold with data rather than guesswork, and read the results in a free confusion table that runs in your browser.
⚡ In short
- A bot that works on the questions you wrote is not tested. Test it on questions real customers wrote.
- Label each test question with the intent it should trigger, including ‘fallback’ for questions the bot shouldn’t answer.
- Count confident wrong answers separately from safe misses. They are not equally bad.
- Put the test set into pytest so every change is checked, and pick your threshold by sweeping it.

Why test
Your bot passes your own questions by definition
When you build an intent matcher, whether it uses keyword rules, difflib similarity, TF-IDF, embeddings or an AI model behind an API, you write example phrases for each intent. Then you try a few questions, the right answers come back, and it feels finished.
The trouble is that you tested the bot with the same kind of phrases you trained it on. Customers don’t write like that. They write “cod avlbl?”, “parcel not come yet”, “my money back pls” and “is there someone real here”. Some of their questions match two intents at once. Some don’t belong to any intent and should get an honest “I’m not sure, let me get a person”.
A test set fixes this. It is a file of real questions, each labelled with the intent it should trigger. You run the bot over it and count the results. It takes an afternoon to build the first version, and after that every change to your matcher, your phrases or your threshold can be checked in seconds.
If you haven’t built the bot yet, start with our Python tutorial, which goes from a 20-line keyword bot to grounded AI answers. This post assumes you already have a function that takes a question and returns an intent name.
The test set
Build a labelled CSV from real questions
Your test set should come from real customer messages, not from your imagination. Look in your WhatsApp business chats, support email, contact form submissions and, if you already run a website chat, the chat history. Copy questions exactly as written, typos included. Remove names, phone numbers and order numbers first.
Aim for at least five test questions per intent, and more for the intents that matter most. A small shop bot with ten intents needs about 50 to 100 rows to start. Then add a group of questions that should go to fallback: off-topic messages, things the bot must not handle (complaints that need a person, medical or legal questions), and plain nonsense. A bot that answers “what’s the weather” with your return policy has a real problem, and only fallback tests will show it.
Two rules keep the set honest. First, don’t reuse training phrases as test questions; keep the files separate. Second, label before you run the bot, so you aren’t tempted to accept whatever it said. If two people disagree on a label, the intents probably overlap, which is useful to know in itself.
Save it as a CSV with two columns, question and expected:
question,expected
what time do you open,hours
are you open on sunday,hours
how much is delivery,delivery_cost
delivery charges to pune?,delivery_cost
when will my order arrive,delivery_time
can i return this,returns
i want my money back,returns
do you take upi,payment
cod available?,payment
can i talk to a person,human
whats the weather today,fallback
Scoring it
A twenty-line evaluate script
With a test set in place, scoring takes a few lines of standard-library Python. The script below assumes your bot exposes a predict_intent(question) function that returns an intent name, or “fallback” when it isn’t confident.
It reports three numbers that matter more than a single accuracy score:
- Accuracy: how many questions got the expected intent.
- Confident wrong answers: the bot picked a real intent, but the wrong one. The customer gets a wrong answer delivered with confidence. This is the worst outcome.
- Safe misses: the bot said it wasn’t sure. The customer is handed to a person. Annoying, but harmless.
It also prints the most common confusions, such as “expected delivery_time, got delivery_cost”, which tell you exactly where to work. Paste its output, or your own expected and predicted pairs, into the free tool below to see precision and recall per intent.
# evaluate.py: run every test question through your bot and score it
import csv
from collections import Counter
from bot import predict_intent # your function: question -> intent name or "fallback"
def evaluate(path="test_set.csv"):
rows = list(csv.DictReader(open(path, encoding="utf-8")))
pairs = [(r["expected"], predict_intent(r["question"])) for r in rows]
correct = sum(1 for e, p in pairs if e == p)
wrong = [(e, p) for e, p in pairs if e != p and p != "fallback"]
missed = [(e, p) for e, p in pairs if e != p and p == "fallback"]
print(f"accuracy {correct}/{len(pairs)} = {correct / len(pairs):.0%}")
print(f"wrong answers (worst kind): {len(wrong)}")
print(f"safe misses (bot said it was not sure): {len(missed)}")
print("most common confusions:")
for (e, p), n in Counter(wrong).most_common(5):
print(f" expected {e}, got {p}: {n}x")
return pairs
if __name__ == "__main__":
evaluate()
Free tool
Turn your results into a confusion table
Paste one result per line as expected, predicted. The table shows accuracy, wrong answers versus safe misses, precision and recall for each intent, and the confusions to fix first.
Paste your results, one per line: expected intent, predicted intent (comma, tab or -> between them). Use fallback when the bot said it was not sure.
Runs only in your browser; nothing is sent or saved. Paste the output of evaluate.py or type the results by hand.
Reading the numbers
Precision, recall and what to do about each
The confusion table gives two numbers per intent. They answer different questions, and each points to a different fix.
Recall: does the bot recognise this intent?
Recall is the share of questions that should trigger an intent and actually did. Low recall on "returns" means customers asking about returns often get fallback or another intent. Fix it by adding more example phrases from real chats, especially short ones, slang and common misspellings.
Precision: when the bot says this intent, is it right?
Precision is the share of the bot's picks for an intent that were correct. Low precision on "delivery_cost" means other questions are being pulled into it. Fix it by making the competing intents' phrases more distinct, or by adding negative rules (for example, "when" questions are about time, not cost).
Pairs that keep swapping
If two intents keep getting confused in both directions, like delivery_time and delivery_cost, ask whether customers really see them as different. Often the best fix is one combined answer: "Delivery is ₹60 (free over ₹999) and takes 2 to 4 days." Merging intents is a legitimate fix, not a defeat.
Fallback precision
If fallback has low precision, the bot is too cautious and hands over questions it could answer. If real intents have many fallback errors, it may be worth lowering the threshold, which the sweep below tests properly.
pytest
Make every change prove itself
Once the test set exists, wire it into pytest. The file below creates one test per row, so a failure names the exact question that broke, plus one summary test that fails if more than two questions get a confident wrong answer.
Run it before you change phrases, swap similarity methods, upgrade a library or change the threshold. A chatbot python project without this tends to get worse quietly: a phrase added to fix one question breaks two others, and nobody notices until a customer does.
Two practical tips. Don't expect every row to pass. Real test sets include hard questions, and demanding 100% pushes you to overfit. Instead, track the pass count over time and treat a drop as a failure. Second, keep a small "never again" group: every time a real customer gets a wrong answer in production, add that exact question to the CSV with its correct label. Over months, this becomes your most valuable file.
If your bot calls an AI model, results can vary slightly between runs. Run the AI tests a few times and look at the pass rate rather than a single run, and keep costs in mind; our API cost guide explains what each request costs.
# test_bot.py: run with pytest -q
import csv
import pytest
from bot import predict_intent
ROWS = list(csv.DictReader(open("test_set.csv", encoding="utf-8")))
@pytest.mark.parametrize("row", ROWS, ids=[r["question"] for r in ROWS])
def test_intent(row):
assert predict_intent(row["question"]) == row["expected"]
def test_no_more_than_two_wrong_answers():
wrong = 0
for r in ROWS:
got = predict_intent(r["question"])
if got != r["expected"] and got != "fallback":
wrong += 1
assert wrong <= 2, f"{wrong} questions got a confident wrong answer"
Choosing a threshold
Sweep it instead of guessing
Most matchers produce a score, and you pick a cutoff below which the bot says "not sure". Tutorials often use 0.5 or 0.6 without explanation. The right value depends on your intents and your customers, and you can find it with a simple sweep.
The script below tries six cutoffs and prints right, unsure and wrong for each. As the cutoff rises, wrong answers fall and unsure answers rise. Pick the point where wrong answers are acceptably low, usually close to zero, without sending half your questions to fallback.
For a small business bot, a confident wrong answer usually costs more than a handover. A customer told the wrong return window may lose money or trust; a customer handed to a person just waits a little. So err towards the higher cutoff, and make sure the handover is good: a person replying quickly in business hours, and an away form outside them.
Re-run the sweep whenever you add many phrases or change the matching method, because the best cutoff moves.
# sweep.py: find the confidence threshold that balances answers and mistakes
from bot import score_intents # your function: question -> list of (intent, score)
import csv
rows = list(csv.DictReader(open("test_set.csv", encoding="utf-8")))
for cutoff in [0.3, 0.4, 0.5, 0.6, 0.7, 0.8]:
right = wrong = unsure = 0
for r in rows:
intent, score = max(score_intents(r["question"]), key=lambda x: x[1])
if score < cutoff:
intent = "fallback"
if intent == r["expected"]:
right += 1
elif intent == "fallback":
unsure += 1
else:
wrong += 1
print(f"cutoff {cutoff}: right {right}, unsure {unsure}, wrong {wrong}")
Common traps
Five chatbot Python testing mistakes that make scores lie
- Testing on training phrases. Scores look excellent and mean nothing. Keep the two files separate, and ideally have someone else write or collect the test questions.
- Too few tests per intent. With two tests, one mistake moves recall by 50 points. The tool flags intents with fewer than three tests for this reason.
- No fallback questions. Without them you can't see whether the bot answers things it shouldn't. Include at least 10% off-topic or must-not-answer questions.
- Only clean English. If your customers write in Hinglish or Tamil script, your test set must too. Our Indian-language test pack has ideas.
- Fixing the test instead of the bot. If a label looks wrong, check with someone else before changing it. Relabelling until the score rises is easy and pointless.
A sixth, quieter trap is letting the test set go stale. Add new real questions every month, especially ones from new products, new policies or seasonal events.
Hosted bots
The same method works without your own code
You don't need to write a matcher to use this method. If you run a hosted bot, the test set and the confusion table still apply; you just run the questions by hand or with a colleague and type the results.
With LiveChatBot, the free keyword bot answers from your saved replies, and the AI Agent answers from the knowledge you give it. Take your CSV, ask each question in the chat widget on your site, and note which saved reply or answer came back. Write the pairs into the tool above. It's slower than pytest, but it tells you the same things: where the bot confuses topics, which topics need more wording, and whether it answers questions it shouldn't.
Many developers end up with a mix: a hosted widget, inbox and Android app for the team, plus some Python of their own around it, such as scripts that process ticket webhooks or call the REST API. If you need technical details for that kind of integration, see the integrations page or ask our team for developer documentation. The golden-question testing guide shows how to keep a manual test list for a hosted bot.
Getting started
An afternoon to a tested bot
Collect 50 real questions and label them, including some for fallback. Save them as test_set.csv. Copy the evaluate script, point it at your predict_intent function, and run it. Paste the output into the confusion table and fix the top two confusions. Add the pytest file and run it after every change. When you've changed phrases significantly, run the sweep and update your threshold.
That loop of test, fix and retest is the difference between a chatbot python demo and a bot you can put in front of customers. And if you'd rather spend your time on the integration than on the inbox, the widget and the phone app, you can use a hosted bot from livechatbot.in; the free plan covers two websites with 100 visitors per website each month. See pricing for packs.
Frequently asked questions
How many test questions does a chatbot python project need?
Start with at least five per intent plus a group of fallback questions, often 50 to 100 rows for a small bot. Add real questions every month.
What is the difference between precision and recall?
Recall asks whether the bot recognises an intent when it should. Precision asks whether it is right when it picks that intent.
Why separate wrong answers from fallbacks?
A confident wrong answer misleads the customer. A fallback just hands them to a person. Most businesses should accept more fallbacks to get fewer wrong answers.
Should every pytest test pass?
Not necessarily. Track the pass count and the number of confident wrong answers, and fail the build when either gets worse.
Can I test a hosted bot this way?
Yes. Ask each test question in the widget, note what came back and paste the pairs into the confusion table.
Keep reading
Related guides
Chatbot Project Plan for a Business: Work Back From Your Go-Live Date, Task by Task (Free Backward Scheduler)
Chatbot Price Quotes, Side by Side: Turn Three Different Offers Into One 24-Month Number
Test your bot on real questions
Label real customer questions, score the results and fix the top confusions first. Start free on livechatbot.in.