Green scoreboards are comforting. They are also how many GCC teams ship systems that look ready in English demos and fail on Arabic tickets, local policies, and real tool use.
The industry is finally talking about an evaluation gap: coverage on public benchmarks is not the same as reality alignment in production. For an Omani or wider Gulf enterprise, the lie is sharper still — because the default eval set was never built for your language, your documents, or your approval path.
Here is how to stop letting English leaderboards decide whether your agent or assistant is fit for work.
What an eval set is supposed to do
An evaluation set is not a trophy. It is a falsification instrument. Its job is to catch the failures you care about before customers, auditors, or operators do.
That means the set must look like:
- the questions users actually ask
- the documents the system must ground in
- the tools the agent may call
- the languages and registers in play
- the places a human must stop the system
If your set only measures fluent answers to clean English prompts, it is not measuring production. It is measuring a demo you already knew how to pass.
The English scoreboard problem
Public coding and reasoning boards are useful for model vendors. They are weak as a sole gate for enterprise Arabic systems for simple reasons:
- they under-represent Gulf dialect, mixed Arabic–English, and noisy speech-to-text
- they rarely score retrieval honesty against your policies and contracts
- they almost never score tool permission mistakes — calling the wrong system, with the wrong identity
- they ignore human-gate behaviour: when the model should refuse, escalate, or wait
A model can look excellent on a board and still invent a clause from the wrong PDF, answer a Muscat support call in stiff formal Arabic that users reject, or take an irreversible step because nothing in the eval forced a stop.
Board green is not production green. Treat boards as ingredient checks. Treat in-house evals as the product check.
Reality alignment beats coverage theatre
Teams often respond to pressure by adding more test cases of the same kind: longer English prompts, more multiple-choice items, more generic safety questions. That is coverage theatre. It increases the number of ticks without changing what is being measured.
Reality alignment asks a harder question: does this system behave correctly on the work we actually run?
For a GCC deployment, alignment usually includes:
- Arabic MSA on record and policy tasks
- dialect or mixed register on support and voice
- retrieval over scanned, mixed, and local documents
- agent steps that touch identity-bound tools
- residency-safe logging of traces used for review
If those are absent from the eval, the eval is not strict. It is polite.
Build four thin packs, not one fat average
One blended “AI quality” score will hide channel failure. Prefer small, named packs:
- User language pack — real support intents in the register customers use; include ASR noise if voice is in scope.
- Record pack — MSA questions grounded in your policies, contracts, and circulars; score citation honesty, not style.
- Agent tool pack — multi-step tasks with allow-listed tools; score wrong-tool use, over-privilege attempts, and stop behaviour.
- Gate pack — cases that must escalate or wait for a human; a “helpful” answer here is a fail.
Keep each pack small enough that a domain owner can review every item. A hundred honest cases beat a thousand synthetic ones nobody trusts.
Do not average the packs into a single vanity number for the steering committee. Report them separately. A system that aces English coding items and fails the record pack is not “mostly ready.”
What to score (qualitatively rigorous, not fake-precise)
You do not need invented industry percentages. You need rubrics operators can apply:
- Intent correct? Did we understand the job?
- Grounded? Did claims map to retrieved sources the user is allowed to see?
- Register fit? Did tone match the channel (support vs formal record)?
- Tool discipline? Did the agent stay inside allow-lists and least privilege?
- Gate discipline? Did irreversible steps wait for the right human?
- Refusal quality? When the system should not answer, did it stop cleanly?
Score with humans who own the process, not only with automatic similarity to a reference paragraph. Automatic metrics help triage. They do not replace judgment on policy and safety.
How eval sets start lying
Watch for these patterns:
- The golden demo set — only cases marketing already cleared.
- The translated set — English items machine-rendered into Arabic that no customer would type.
- The MSA-only set — fine for contracts, silent on contact-centre reality.
- The chat-only set — never forces a tool call or an approval gate.
- The frozen set — never updated when policies, products, or fraud patterns change.
- The vendor set — optimised to make a purchased model look good.
Each of these produces confidence. None of them produce truth.
A practical build sequence for GCC teams
- List production surfaces — support, employee help, policy Q&A, agentic back-office, drafting.
- For each surface, collect real traces — anonymised tickets, redacted policies, failed pilot chats. Prefer ugly truth over clean fiction.
- Label register and risk — dialect vs MSA; read-only vs write; reversible vs irreversible.
- Write expected behaviour, not only expected prose — including “must escalate” and “must not call tool X.”
- Split packs so a failure is diagnosable.
- Run the pack on every material change — model swap, prompt change, retrieval index refresh, new tool.
- Retire and refresh items when the business changes. Stale green is a lie with a date.
This sequence is product work. It is not a research side quest.
Agents make weak evals more dangerous
Assistants that only draft text fail in public. Agents that call tools fail in systems of record. That is why agent evaluation cannot stop at “does the answer sound right?”
For agents, the eval must include:
- trajectory checks — did it take a senseless detour or a forbidden shortcut?
- identity checks — which credentials were used? shared service accounts are a design smell.
- state checks — what changed in the external system, and can you reverse it?
- human path checks — was the approval screen shown with enough context to decide?
If your agent eval never creates a fake write, never denies a tool, and never requires an approver, you are evaluating a chatbot costume.
What “good enough to ship” should mean
Shipping criteria should sound like this:
- language and record packs pass at a bar your domain owners set
- tool pack shows no silent over-privilege in the tested scenarios
- gate pack shows mandatory stops actually stop
- residual failures have owners, not slideware
“We outperform model X on public board Y” can be a vendor conversation. It should not be the go-live sentence for a system that touches Omani customer data or internal ERP.
The quiet fix
Your eval set is lying when it is optimised for comfort, procurement theatre, or vendor comparison — instead of for the conversations, documents, and actions your organisation actually runs.
Build thin, local, bilingual-aware packs. Score retrieval, tools, and gates as first-class. Keep English boards in their place as ingredient tests.
When the in-house set turns red, that is not embarrassment. That is the system finally telling you the truth — early enough to act.