What it takes to benchmark Arabic AI fairly

Arabic is spoken by more than 400 million people, and new models that claim to handle it arrive every month. Whether a claim holds depends on how the test behind it was built: who collected the data, who labeled it, which Arabic it covers and how the answers were scored. Those choices matter more than any single ranking.

aixplain has worked on those choices since 2022: building Arabic speech technology and data with SDAIA, helping found BALSAM, the region's shared benchmark for Arabic language models, and running an independent human evaluation of ALLaM for HUMAIN. This article sets out what we learned, including from the public leaderboard we ran.

Why the test should come from a third party

Whoever designs a test decides what it measures. A model maker that grades its own model picks the tasks, the data and the scoring rules, and even in good faith those choices lean toward what the model was built to do.

Public test data has a second problem: it leaks. Data posted online can end up in a model's training set, so a model can score well on questions it has effectively seen before. Scores then overstate progress.

A fair test needs four things:

• Hidden test data. Development examples can be open, so teams can try their models, while the test set stays private. BALSAM works this way.
• The same conditions for every model. The same prompts, settings and scoring, with every answer collected through one interface.
• Blind judgment. Reviewers see the answers without knowing which model wrote them.
• A published method. Others can check how the scores were made. BALSAM's design and findings appeared in a peer-reviewed paper at ArabicNLP.

aixplain runs agents on any model, so it has no stake in which one wins. That independence is what a model maker gets from an outside evaluation: the prompts, the anonymized answers and the reviewers all come from outside the team that built the model.

Start with the data: which Arabic, and whose voices

Arabic is several languages in daily use. Modern Standard Arabic fills news, law and government. People speak dialects, Gulf, Levantine, Egyptian, Maghrebi and more, each with its own words and sounds. They mix in English mid-sentence, write dialect in Arabic or Latin letters, and write numbers in Arabic-Indic or Western digits. A benchmark can only speak for the Arabic in its data.

Our speech benchmark tested six recognition systems on five dialects: Chami, Egyptian, Hijazi, Najdi and Emirati. Building it showed how much the details of a dataset decide what it can tell you.

• Who is speaking. Women were 17% of the speakers, from 10% of the Chami and Hijazi recordings to 28% of the Emirati ones, and the systems made slightly more errors on their speech. A test set that underrepresents a group can't show how well a system serves it.
• Where they are speaking. About half the recordings were clean. The rest had music or noise behind the voice, and the average word error rate rose from 49% on clean audio to 61% with noise.
• What is known about them. For about one recording in six, the speaker's age, gender and accent were unknown. Without that information, a gap in the results can't be traced to its cause.

Our March 2026 report on four dialects shows the same pattern: about seven in ten speakers were men, and nearly half the audio had music, noise or a distant microphone in it.

For an organization choosing a model, the best test data is its own: its calls, documents and prompts, collected and labeled with review and quality checks at every step.

Annotation: native speakers, written rules, measured agreement

Labels are the answer key, and a benchmark inherits every mistake in them. Arabic makes labeling harder than it looks.

• Transcripts written from scratch. In our speech benchmarks, people transcribed every recording by hand, with no speech recognition system's draft to correct, so no system's habits shaped the answer key.
• Native speakers of the dialect. A Najdi recording is transcribed and checked by people who speak Najdi. A fluent speaker of another dialect will miss some words and misread others.
• Written rules for spelling. Dialects have no standard spelling, so the same word can be written several ways, all of them right. Guidelines settle how to write dialect words, English words, numbers, hesitations and diacritics, so every annotator writes the same thing the same way.
• Agreement, measured. Annotators label overlapping samples, and their agreement is tracked against a target set before the work begins. Low agreement points to unclear guidelines, and those get fixed before the labels are used.
• Blind, structured review for language models. In our human evaluations, bilingual reviewers compare anonymized answers in pairs. Each gives a rubric score, tags failures such as hallucination, inconsistency, verbosity or formatting, states a preference and how strong it is, and writes a short reason.

Metrics: score Arabic the way Arabic is written

Word error rate, the standard score for speech recognition, counts the words in a transcript that differ from the reference. In Arabic, many of those differences are matters of spelling:

• Alef written with or without hamza (أ إ آ ا)
• Taa marbuta and haa at the end of a word (ة ه)
• Alef maqsura and yaa (ى ي)
• Diacritics, which most writing leaves out
• The elongation mark, tatweel, used for style
• Digits written as ٣ or 3, or a number written out as a word

Normalizing the text before scoring removes those false errors. The rules should be published with the results, because two benchmarks with different rules give the same system different scores.

Dialects strain the metric further. A reference transcript fixes one spelling for words that have several, so a system can be marked wrong for a spelling any native reader accepts.

Our March 2026 speech report added a second score for this, edited word error rate. A language model rewrote each system's transcript to follow the reference's spelling, changing a word only when the meaning stayed the same and the difference was spelling alone. People then reviewed a sample of the rewrites. In the Najdi example below, the system wrote numbers as digits and spelled dialect words its own way. Once the spelling matched, its word error rate fell from 43.6% to 9.1%.

Text has the same problem. BALSAM's authors compared common ways of scoring Arabic writing with what human judges thought of it. Metrics that count overlapping words, such as BLEU, ROUGE and BERTScore, agreed poorly with the people. A large language model acting as judge agreed closely, more closely with the judges' average than the judges agreed with each other. Our own leaderboard scored language models with ROUGE-L and BLEU, and that finding is why we pair automatic metrics with model judges and human review.

Report every dialect separately

An average across dialects hides the differences that decide whether a system works for your users. On our speech benchmark, rankings changed from one dialect to the next:

• One system was among the two most accurate on Hijazi, Najdi and Emirati speech, and in the bottom half on Chami and Egyptian.
• Another was the second most accurate on Chami and the second least accurate on Hijazi.
• Egyptian was the hardest dialect overall, with an average word error rate of 60% against about 51% for the other four.

A team serving callers in Cairo and a team serving callers in Riyadh could reach opposite conclusions from the same benchmark, and both would be right. Report each dialect, each audio condition and each group of speakers on its own, next to cost and speed.

From a leaderboard to a method

Our public leaderboard published interactive results for 34 Arabic language models on 11 groups of tasks, last updated in August 2026, and for six speech recognition systems on five dialects, last updated in July 2026. It grew out of the reports we published before it, on dialectal Arabic speech recognition and on Arabic language models across many tasks.

A leaderboard is a snapshot. Models change every few weeks, and a general ranking says little about how a model will do on your data, in your dialect, under your conditions. The method lasts longer than any ranking, so we've written it down here. For a public standard on Arabic language models, see BALSAM's leaderboard.

Benchmark the models you're considering, on your own data

Benchmarking is a core aixplain service, and it follows the method above:

• Your data. Your own prompts, documents or recordings, or public datasets, in Arabic and English.
• Automatic metrics built for Arabic. ROUGE and BLEU for text. Word, character and match error rates for speech, with the text normalized first.
• Model judges and human evaluation. Rubric scores, pairwise preferences, issue tags and reviewer agreement, for what metrics can't see.
• Cost and speed. Cost per call, latency and the share of failed calls for every model, next to its quality.

The aixplain OS runs on any model, including Arabic-first models such as ALLaM and Jais, so the model that wins your benchmark is the one your agents use. Evaluator, coming soon, brings the same scoring to agents: it scores an agent's runs over time, so a new model or version has to show it does better.

See aixplain at work

aixplain is the operating system for accountable AI agents you own. Organizations run more of their operations and services on agents, on any model and any server. Every run is checked against policy, and every agent answers to an owner.

Book a demo to benchmark the models you're considering, in Arabic and English, on your own data.