AI Based Chat Fact-Check: How to Catch Answers Your Bot Made Up, With a Free Grounding Checker

AI based chat assistants write fluent answers, and that’s exactly why their mistakes are hard to spot. A wrong delivery time or an invented return policy reads just as confidently as a correct one. You won’t find these errors by asking the bot a few test questions; you find them by comparing what it said with what you actually told it. This guide explains the five kinds of made-up answers, a 20-minute weekly fact-check routine, how to rewrite your knowledge so the gaps close, and what to say to a customer who was given wrong information. The checker compares any answer with your knowledge text and flags what isn’t supported.

👤 What the visitor seesDelivery question · checked
do u deliver to goa? how fast
🤖 Before the fixYes! We deliver to Goa the same day, guaranteed.
❌ ‘Goa’ and ‘same day’ are not in the shop’s knowledge
🤖 After the fix · from “Delivery FAQ”We deliver across India in 3 to 5 working days. For Goa, our team can confirm the exact date.

Illustrative chat for a made-up shop.

⚡ In short

  • Made-up answers fall into five types: wrong numbers, invented names, over-promises, merged policies and stale facts.
  • They’re easiest to catch by comparing the answer with your knowledge text, not by reading the answer alone.
  • Most gaps close when you add explicit ‘we don’t’ statements and exact numbers to your knowledge.
  • Paste any bot answer and your knowledge into the checker to flag unsupported numbers, names and promises.
Ai based chat in LiveChatBot: AI Agent knowledge sources and a sample chat answered from them
How it looks in LiveChatBot: AI Agent knowledge sources and a sample chat answered from them.

Why it happens

Why an assistant fills gaps instead of saying “I don’t know”

A language model writes the most likely next words given what came before. When your knowledge clearly answers the question, the most likely words are your words. When it doesn’t, the model still produces likely-sounding words, and in a delivery conversation, “Yes, we deliver there in 2 days” is very likely-sounding.

Three situations make this worse. The question is close to something your knowledge covers, so the bot stretches a nearby fact: you deliver to Pune in 2 days, so it says the same for Goa. The customer pushes: “are you sure? I need it by Friday.” Or the knowledge is silent on a common topic, such as weekends, and the bot fills the gap with what most shops do.

Good systems reduce this by answering only from the knowledge you give them and by being told to say when the answer isn’t there. In LiveChatBot, the AI Agent answers from the knowledge you provide. But no setting replaces the habit of checking. A sample of real conversations, compared line by line with your knowledge, is the only reliable way to know your AI based chat is telling the truth. The sections below show how to do that quickly.

Five types

The kinds of made-up answers to look for

Each with an illustrative example from a made-up shop.

🔢

1 · Wrong numbers

Knowledge says COD up to ₹5,000; the bot says ₹10,000. Prices, limits, days and percentages are the most common and most costly errors.

🏷️

2 · Invented names

The bot mentions a city you don’t serve, a product you don’t sell or a staff member who doesn’t exist.

📣

3 · Over-promises

“Always free”, “guaranteed”, “same day”. Words your policies never use, which customers will quote back to you.

🔀

4 · Merged policies

Two true facts combined wrongly: free delivery above ₹999 plus delivery to Pune becomes “free delivery to Pune”.

📅

5 · Stale facts

The answer matches last year’s price list, because an old page or document is still in the knowledge.

Free tool

Check an answer against your knowledge

Paste your FAQ or policy text and one bot answer. The checker flags numbers, names and risky promises that your text doesn’t support.

Runs in your browser; nothing is saved or sent. It checks numbers, names and risky words against your text. It can’t judge meaning, so a “found” number can still be used wrongly.

How the checker works

What it catches, and what it can’t

The checker does what a careful person does on a first pass, only faster.

Numbers. Every number in the answer is looked up in your knowledge text. “₹10,000” isn’t there; “2” is. Commas are ignored, so 5,000 and 5000 match.

Names. Capitalised words that aren’t at the start of a sentence are treated as names: places, products, people. If “Goa” isn’t anywhere in your knowledge, the bot didn’t get it from you.

Risky words. Words like always, never, guaranteed, free, same day and unlimited are flagged when your knowledge doesn’t use them. These are the words customers screenshot.

What it can’t do: understand meaning. If your knowledge says “returns within 7 days” and the bot says “exchanges within 7 days”, the checker sees 7 in both and passes it. If the bot says “2 days” for Goa, 2 appears in your Pune line, so the number passes, but the name Goa doesn’t. That’s why the checker is a first pass. It points your eyes at the risky parts; you still read the answer for meaning.

Trap questions

Questions that tempt a bot to guess

Ask these on purpose before launch and after every knowledge change. The right answer is often “I’ll check with the team”.

Do you deliver to Port Blair?
Can I pick up from your Hyderabad store?
Is delivery to Nepal possible?
I need it by tomorrow, can you guarantee that?
Is same-day delivery available if I pay extra?
Will it reach before Diwali for sure?
Can I pay COD for a ₹25,000 order?
Can I return a sale item after 10 days?
Is there a student discount?
Do you have the blue XL version of this?
Is the premium plan cheaper yearly?
Do you sell gift cards?

Change the details to fit your business: use a city you don’t serve, a limit just above yours and a product you don’t sell. A well-grounded AI based chat assistant should decline or hand over on every one of them. If it confidently says yes, you’ve found a gap in your knowledge before a customer did. Write the missing “we don’t” line, then ask again in a different wording to confirm the fix.

Weekly routine

A 20-minute fact-check every week

You don’t need to check every conversation. A small, regular sample catches patterns early.

  1. Pick ten AI answers from the past week: five from your most common topics, five random. Include at least two from evenings or weekends, when nobody was watching.
  2. Run each through the checker with the knowledge section that should cover it.
  3. Read the flagged ones for meaning. Was the number wrong, or just written differently?
  4. Write down each real error in a simple log: date, question, wrong answer, correct answer, cause (gap, stale page, merged facts).
  5. Fix the cause, not the answer. Usually that’s one or two sentences added to your knowledge.
  6. Ask the same question again after the fix, in two different wordings.

Ten answers a week is about 500 a year, enough to catch nearly every recurring problem. If you find more than two real errors in a week, double the sample for the next month. Our regression testing guide shows how to turn fixed questions into a re-test list.

Closing the gaps

Rewrite your knowledge so the bot has less to guess

Most made-up answers trace back to something your knowledge doesn’t say. Four rewrites fix the majority.

Say what you don’t do

“We deliver across India” invites “yes” to every city. Add “We don’t offer same-day delivery” and “Delivery to the Andaman and Nicobar Islands and Lakshadweep is not available.” Negative statements are some of the most useful lines in a knowledge base.

Put limits next to every offer

“Free delivery” is dangerous alone. “Free delivery on orders above ₹999; below that, delivery costs ₹79” isn’t.

One fact, one place

If your delivery time appears on three pages with three numbers, the bot will pick one. Keep each fact in one place and delete the old copies.

Tell it what to do when unsure

Your instructions should say something like: “If the knowledge doesn’t answer the question, say you’ll check with the team, and offer to connect the customer.” Then test it with a question you know isn’t covered.

For a deeper review of which pages to include, see our knowledge audit, and for sentence-level tips, how to write answers your bot can use.

Before and after

Knowledge lines that caused errors, rewritten

Illustrative examples.

Delivery
🚩 BeforeWe deliver all over India quickly.
✅ AfterWe deliver across India in 3 to 5 working days. Pune, Mumbai and Nashik: 2 days. No same-day delivery.
Free delivery
🚩 BeforeEnjoy free delivery!
✅ AfterFree delivery on orders above ₹999. Below ₹999, delivery costs ₹79.
Returns
🚩 BeforeEasy returns.
✅ AfterUnused items can be returned within 7 days of delivery. Innerwear and sale items can’t be returned.
Weekends
🚩 Before(nothing)
✅ AfterWe’re closed on Sundays. Orders placed on Sunday ship on Monday.

When a customer was told something wrong

What to do after the bot made a promise you can’t keep

Sooner or later, a customer will quote a wrong answer back to you: “Your chat said free delivery to Goa.” How you handle it matters more than the error itself.

Check the transcript first. Read exactly what was said. Sometimes the customer misread a correct answer.

If the bot was wrong, own it. “You’re right, our chat assistant gave you the wrong information, and I’m sorry.” Don’t blame “the AI” as if it were someone else’s system. It’s yours.

Decide whether to honour it. For small amounts, honouring the wrong answer once is usually cheaper than losing the customer and their goodwill. For large amounts, explain the correct policy clearly and offer something reasonable.

Fix the knowledge the same day, and add the question to your weekly re-test list.

Our guide to handling angry customers covers the conversation itself. The lesson for your AI based chat setup is simple: every error a customer finds is one your weekly check missed, so add it to the routine.

Getting started

Run your first fact-check today

Open your inbox, pick five recent AI answers, and paste each into the checker with your knowledge text. Fix the first gap you find. Then put a 20-minute weekly slot in your calendar.

LiveChatBot’s AI Agent answers from the knowledge you give it, every conversation is saved in the team inbox for review, and when AI replies aren’t available the free keyword bot answers from your saved replies. The free plan covers two websites with 100 visitors per website each month. Start on livechatbot.in.

Frequently asked questions

Why does an AI based chat assistant make things up?

It writes likely-sounding answers. When your knowledge doesn’t cover a question, a likely-sounding guess can come out instead of ‘I don’t know’.

How do I check whether my bot’s answers are correct?

Compare a weekly sample of real answers with your knowledge text, looking first at numbers, names and promise words like ‘always’ or ‘guaranteed’.

How many conversations should I check?

About ten a week for a small business, including some from evenings and weekends. Double it for a month if you find more than two errors.

What’s the best way to stop made-up answers?

Ground the bot in your own knowledge, add ‘we don’t’ statements and limits to your text, and tell it to say when it isn’t sure.

What should I do if a customer was given a wrong answer?

Check the transcript, own the mistake, decide whether to honour it, and fix the knowledge the same day.

Keep reading

Related guides

📘

Agent Live Chat Reply Checker: Score Any Message Against 12 Habits Before You Hit Send

Read the guide →

🧭

Advantages of Bots Beyond Chat: 12 Repetitive Website Jobs Software Bots Can Take Off Your Week

Read the guide →

🛠️

Advantages of AI Chatbot Answers vs Their Limits: An Honest Ledger, With the Fix for Each Limit

Read the guide →

Make sure your bot only says what you told it

Ground the AI Agent in your knowledge and check a sample every week. Start free on livechatbot.in.