Arabic AI Chatbots for Saudi Businesses: How to Evaluate a Pilot

Write a short scope before requesting proposals: who will use the assistant, which channel it serves, which questions are included, which documents it can use and what it may change. An information-only pilot can be evaluated without granting permission to edit customer accounts, approve refunds or alter orders. List those excluded actions explicitly.

For each included task, specify the desired output and the fallback when information is missing. For example, answering from an approved support-hours record is different from confirming a live appointment. Require a separate test for any action that depends on another system; a conversational confirmation is not evidence that the action succeeded.

OpenAI's evaluation guide recommends defining an objective, collecting a dataset, choosing metrics, comparing runs and continuing evaluation after changes. It includes typical, edge and adversarial cases and warns against judging by a vague impression that the system works. Those are evaluation-method references, not evidence that a particular model is best for Arabic or that Ting uses it in production.

Our proposed pilot method is to give each candidate the same permitted source material, test cases and access restrictions. Record the model/application version and source version. Keep some independently checked examples out of prompt tuning so the comparison does not merely reward memorizing the demonstration. Use authorized, privacy-appropriate examples; the fictional cases below are a starting pattern, not a substitute for your actual audience's questions.

Do not infer accuracy from labels such as native Arabic, multilingual, translated or fine-tuned. Ask the supplier to explain the proposed pipeline, then compare its outputs against the same expected answers. The sources cited here do not establish a universal winner between those architectures, and this guide does not claim one.

Include formal Arabic, natural Saudi phrasing, spelling variation, a mixed Arabic/English product identifier and a correction in a later turn where relevant to your audience. Score meaning and factual support separately from tone. A fluent answer can still select the wrong branch or invent a price. Add actual approved examples from the intended workflow before setting an acceptance threshold.

The test pack at the end uses one declared fictional fact: support closes at 17:00. No price list, account-edit permission or verified staff connection is provided. The questions and expected behavior are proposed fixtures; we have not executed a chatbot benchmark with them. Extend the context with your own approved records instead of treating the fictional hours as a Ting service promise.

For each run, keep the exact input, permitted context, actual answer, expected behavior, source reference, pass/fail reason, version and reviewer. Track correctness on answerable questions, unsupported answers, incorrect actions, context-correction failures and escalation failures separately. Record the denominator and test mix with any rate. Passing these cases does not demonstrate coverage of every dialect, production reliability or legal compliance.

OWASP's LLM01:2025 describes prompt injection through user input and through external material such as retrieved files or websites. Its mitigation guidance includes constrained behavior, output validation, least privilege, separation of untrusted material and adversarial testing. It does not present retrieval or fine-tuning as a complete defense.

For the pilot, keep account-changing operations disabled unless separately authorized and tested. Treat material retrieved for answering as data, not as permission to run a tool. Test an attempted instruction override inside a document as well as in the user's message. If a proposed action fails or a staff connection is unavailable, require an honest failure response instead of a claim that the action or handoff was completed.

Ask for an itemized proposal covering preparation of the knowledge base, integration, evaluation, channel/provider usage, monitoring and support ownership. Estimate cost from measured usage in the proposed workflow and the actual provider terms. This guide does not give a universal Saudi price or delivery time; scope, data readiness, permissions and integrations need to be known first.

Document where messages and logs are processed, access and retention settings, provider data-use terms and the responsible privacy review before using real data. Do not infer a blanket hosting rule from a generic AI guide. Agree who receives failures, how to stop or roll back the pilot, and which changes require retesting. Use the linked audit, validation and budget resources to define those decisions before a wider rollout.