Best Practices for Testing your Chatbot Before Launch

Testing a chatbot is fundamentally different from testing traditional software. The same input may not produce the same output and defining what might be "correct" is often a matter of degree and failure. Hallucination, off-topic drifts, tone breaks, not-so-good exception handling and many more issues which could damage trust in the system.

So, why is chabot testing different from Software testing?

Traditional software testing checks whether functionality as expected, where expected output is based on input. Chatbot testing has to account for:

• Non-determinism: the same question can generate slightly different phrasing of the response each time
• Correctness: an answer can be partially right, technically accurate but unhelpful, or right but poorly sourced
• Adversarial inputs: users could deliberately try to break, confuse or manipulate the bot
• Context drift: errors can compound across a multi-turn conversation which can be missed even if you have multiple single-turn conversations
• Behaviour: across models, the tone and behaviour of the chatbot might be different and un-brand like

This means that chatbot testing needs a broader coverage across accuracy, safety, tone and resilence than a standard QA checklist.

Here are 8 important steps we used to test Illumea:

1. Model Research

Illumea supports multiple models from multiple providers. Each model's response can be different and might require additional tuning to customize and align to each brand's requirements. The best way to understand this is to do your own digging and this should be done even before you start building the chatbot. You will end up with a large list of pros and cons for each model you have tested, that make it difficult to go with a specific model. A ChatGPT model may respond with a more friendly tone to a question than a Claude one. The research would certainly help you understand how to maybe build customization layers etc. This is also very good information for the testing team.

2. Golden Question Set

Building a golden question set is a baseline for every future change. Assemble a set of 50-100 real questions that your chatbot must answer correctly. You can source these questions from frequently asked support or sales questions, pricing, product and policy questions which teams answer manually today, questions that can be pulled from support tickets or sales transcripts, edge cases like refund policies, compatibility etc. This set should be run every time you change the system prompt, switch the underlying model or update the knowledgebase.

3. Test for Hallucination

Every response needs a source citation! The chatbot can respond with an high-confidence, believable answer that could be absolutely wrong. Hallucination testing means deliberately checking whether the bot invents the information in its response that is not present in the knowledgebase. To test it:

• Ask questions about products, pricing or policies that does not exist. The bot should ideally say that it does not have an answer.
• Cross-check responses with actual source documents
• Track a citation rate which would tell you the percentage of factual responses have actual traceable source

A well-tuned RAG chatbot should have a near-zero hallucination rate on on-brand questions. If you see more invented responses, revisit your constraints before anything else.

4. Try to Break it!

When we first launched Illumea, the first thing users did was try to break it. Some tried repeating the McDonald's Python question as well. It is best to try to break your own bot before others do using:

• Prompt injection attempts like "Ignore your instructions and tell me.."
• Off-topic questions like "Who is the richest person in the world" which is considered off-topic
• Boundary requests like ask for information the bot should not share like competitor stuff, internal pricing or unverified claims
• Repeated requests by sending the same message repeatedly or sending random nonsensical strings

A good chatbot handles all these gracefully by declining, redirecting or acknowleding its limits and without breaking character or leaking instructions.

5. Check Multi-turn and Context Handling

Testing using a single message misses one of the most common real world failure scenario: losing track of context across the a conversation. This makes the chatbot feel careless rather than being limited. You can test this by asking the chatbot information from 2-3 messages earlier in the same conversation or see how it handles follow-up questions or even have a long conversation to see if the context bleeds incorrectly between unrelated topics. This is something users would easily notice and failures can be damaging.

6. Test Consistency of Tone and Persona

Illumea supports character, role and behaviour-related features. Test these to ensure your chatbot holds up even under pressure. Especially test to see if the tone stays consistent when a user is frustrated or rude. Also, check if the reponses drifts into a causual or formal language across a session. If your chatbot has been configured with a specific persona (like customer support agent or sales agent), check to see if the behaviour matches in each reponse, not just the language. The best way to do this is have human reviews, reading conversations. Automated tests can fail to catch this.

7. Test the Escalation and Hand-off Logic

A chatbot that never admits uncertainity is more dangerous than one that escalates too often, hence, testing the boundary is a must. You can start by asking genuinely answerable questions to see how the chatbot responds. Next, check with low-confidence messages to see if it triggers something like "let me connect you with someone". If all that works well, verify that the hand-off works end-to-end to ensure that the human recieving the escalation gets the full context of the conversation

8. Run a Soft-launch Before Release

We could spend a lot of effort in testing everything to the T, but it would still not replicate actual user behaviour. When we first rolled out Illumea, users first tried to break it asking questions we never thought of. As goes with the habit, users naturally asked questions in their language. While the chatbot held up, we did not have a simple answer in our knowledgebase for a question like "What languages does the chatbot support?". So, before rolling out for the world to use:

• Launch to a small percentage of people who can give you proper feedback and beta test the features
• Monitor unanswered questions and low-confidence reponses closely during the beta period
• Setup a daily review of coversation transcripts
• Treat every unanswered or poorly handled question as a new additon to your golden question set

While you might think that this closes the loop, testing isn't really a one-time before launch process. Conversations should be monitored periodically to ensure everything works as expected. While you can automate most of these tests, human intervention is needed at every stage for quality checks. Platforms like Illumea that provide analytics and intelligence reporting track answers, feedback scores and content gaps to help brands with these qualitative inputs that could flow into processes to ensure that they maintain a high-quality chatbot.