Ting Guides · For the COO
How do you evaluate an AI chatbot before launch?
The short answer
Evaluate it on your own traffic, not on a demo script. Build a test set from real customer questions, in the Arabic and English your customers actually write, agree pass marks before testing, then score accuracy, fallbacks, handoff to a person, resistance to manipulation and cost per conversation. Launch only when every must-pass case passes.
Reviewed by Abdullah Al-RashedUpdated
What it depends on
What the bot is allowed to do
Answering from approved content is one test. Taking actions, such as booking, cancelling or changing an order, is a stricter one: each action needs its own failure cases, and the permission check has to sit in your system, not in the model's instructions.
Who writes to it
Customers, staff and suppliers write differently. A test set your team typed in formal Arabic will overstate quality if your customers write in dialect, mix Arabic with English, or type Arabic in Latin letters.
The cost of a wrong answer
A wrong opening time is an annoyance. A wrong price, refund rule or medical instruction is a liability. The higher the cost, the stricter the pass mark and the more topics that go straight to a person.
Where the answers come from
If the bot retrieves from documents, you are testing two things: whether it found the right passage, and whether it answered only from that passage. Score them separately, or you won't know which one to fix.
Channel and volume
A WhatsApp line at peak hours needs reply-time and cost tests under realistic load. A low-volume internal assistant mostly needs accuracy and access tests.
In Saudi operations: dialect, WhatsApp and personal data
General information, not legal advice. Each regulatory point summarises the official source cited beside it, as checked on the date shown under Sources; confirm how it applies to your company with your legal adviser.
Score each way of writing on its own line
Sample test cases by how your customers really write: Gulf dialect, Modern Standard Arabic, Arabizi and mixed Arabic-English. NIST's Generative AI Profile lists performance gaps between languages as a known risk. A single blended score hides a bot that works in English and fails in dialect.
Test across the WhatsApp window
On the WhatsApp Business Platform, free-form replies are only possible inside the 24-hour customer service window that each customer message opens; contacting the customer after it closes needs an approved template. Test what happens when a handoff or a follow-up lands after the window has closed.
Decide where the test data goes
Real conversations carry names, phone numbers and order details. The Personal Data Protection Law allows personal data to leave the Kingdom only for the purposes and under the conditions set in Article 29 and the Transfer Regulation. Mask the test set, or confirm where the model provider and your logs process data, before the first run.[3][4]
Make "I don't know" a must-pass case
SDAIA's generative AI guidelines for the public say that when a publicly available service cannot produce an accurate answer and identify its source, the output should be replaced with a predefined message warning users that the information may not be accurate. Write cases whose answer is not in your content, and require that message rather than a guess.[5]
A launch scorecard for a customer-service bot
ExhibitWorked example
Rasikh Motors, part of the fictional Rasikh Holding, tests its customer-service bot before launch. The pass marks were agreed before the first run, and the decision follows from the rows that block launch, not from an average.
- Policy answers: refunds, delivery, warranty58 pass; 2 promise a refund the policy doesn't allowBlocks launch
- Dialect, Arabizi and mixed language37 correctPass
- Questions with no answer in the content25 fallbacksPass
- Handoff to a person13; 2 lost after the 24-hour windowBlocks launch
- Prompt injection and data leakage0 leaksPass
- Reply time and cost at peak loadWithin bothPass
Launch
| Test group | Cases | Pass mark | Result | Status |
|---|---|---|---|---|
| Policy answers: refunds, delivery, warranty | 60, from past chats | All must pass | 58 pass; 2 promise a refund the policy doesn't allow | Blocks launch |
| Dialect, Arabizi and mixed language | 40 | At least 36 correct | 37 correct | Pass |
| Questions with no answer in the content | 25 | Predefined fallback, no guessing | 25 fallbacks | Pass |
| Handoff to a person | 15 | Each reaches an agent with the conversation attached | 13; 2 lost after the 24-hour window | Blocks launch |
| Prompt injection and data leakage | 30 hostile prompts | No instructions or other customers' data revealed | 0 leaks | Pass |
| Reply time and cost at peak load | 200 simulated chats | Within the agreed limits | Within both | Pass |
The decisionDecision: no-go. Fix the refund answers and the after-window handoff, re-run the two blocking groups, then decide again.
What Ting does and doesn't do here
What we do
- We build the test set from your real conversations, in the mix of Arabic and English your customers use, and agree the pass marks with you before anything runs.
- We test the system whoever built it, including hostile prompts and handoffs, and give you a go or no-go with every failing case and its evidence attached.
- We re-test after the fixes and set up the checks that keep running after launch.
Where Ting helps: AI Solution Evaluation
What we don't do
- We don't issue a regulatory certification. The conclusion covers the agreed cases and scope.
- We don't require a Solution Blueprint first. Evaluation starts from the system you already have.
- We don't hide who built it. When the system under review is one Ting built, the scope and the report say so, and you receive the test cases, data and results so anyone can re-run them.
Sources
Primary sources only, each read in full on the date shown.
- NIST AI 600-1: Generative AI Profile (July 2024)NIST · checked
- WhatsApp Business Platform: pricing and the customer service windowMeta for Developers · checked
- Personal Data Protection Law (Royal Decree M/19 of 9/2/1443H, amended by Royal Decree M/148 of 5/9/1444H)SDAIA · checked
- Regulation on Personal Data Transfer outside the Kingdom, version 2.0 (August 2024)SDAIA · checked
- Generative Artificial Intelligence for Public Guidelines, version 1 (May 2025), §5.3SDAIA · checked
- LLM01:2025 Prompt InjectionOWASP GenAI Security Project · checked
