BALSAM gives Arabic language models a shared standard, with aixplain as a founding member
A consortium of the region's research and government institutions built BALSAM to measure Arabic large language models on blind test sets. aixplain helped found it, and its scientists helped build it.
Arabic is one of the world's major languages, yet large language models still do worse in Arabic than in English. Closing that gap starts with measuring it honestly. BALSAM, short for Benchmarking Arabic LLM Standards and Metrics, is the region's shared answer: a benchmark built by a consortium of academic and government institutions and hosted by the King Salman Global Academy for Arabic Language, which tests Arabic models on questions they have never seen. aixplain is a founding member, and its scientists are among the people who built it.
The challenge: you can't improve what you can't measure
Many Arabic benchmarks reuse public data. Public data can end up in a model's training set, so a model can score well without being good, and scores overstate progress.
Most benchmarks also measure a narrow slice of what a model does, and each lab tended to run its own tests. There was no shared place to compare Arabic models fairly, on the same questions, under the same rules.
“If you can't accurately measure AI performance, you cannot expect it to improve.”
aixplain's part: from founding to the paper
- A founding member. aixplain joined the institutions that founded BALSAM, with SDAIA, MBZUAI, QCRI, NYU Abu Dhabi and others, around the King Salman Global Academy for Arabic Language.
- Science. Dr. Kareem Darwish, aixplain's head of human-AI interaction, helped lead the work. He gave the invited keynote on BALSAM at the ArabicNLP conference, and he is an author of the peer-reviewed paper that describes it, alongside aixplain's Muhammad Elmallah and Abdelrahman El-Sheikh.
- Models from the ecosystem. Model owners can enter a model through any OpenAI-compatible endpoint, or as a public model on aixplain, and BALSAM runs the evaluation.
What BALSAM is
A broad benchmark. Tasks cover the work people ask Arabic models to do: summarization, question answering, information extraction, translation, text classification, creative writing and more, in Modern Standard Arabic and dialects.
Blind test sets. Development examples are open, so teams can try their models. Test sets stay hidden, so no model can be trained on the answers.
A leaderboard. Results for Arabic models sit in one place, under the same rules, so progress can be tracked over time.
A method in the open. The consortium published its design and findings in a peer-reviewed paper at ArabicNLP, so other languages can follow the same approach.
What it found: how you score matters
BALSAM's authors compared common ways of scoring a model's Arabic writing with what human judges thought of it. Metrics that count overlapping words, such as BLEU, ROUGE and BERTScore, agreed poorly with the people. A large language model acting as judge agreed closely, more closely with the judges' average than the judges agreed with each other.
For anyone choosing a model, that's the practical lesson: test on tasks like yours, on data the model hasn't seen, and score it the way people would.
Why it matters: benchmarking before you choose
Benchmarking is a core aixplain service. The aixplain OS runs on any model, so the question every organization faces is which model fits its work, especially in Arabic. BALSAM sets the public standard. aixplain applies the same discipline on an organization's own data: automatic metrics, language models as judges, and human evaluation, side by side with cost and speed.
See aixplain at work
aixplain is the operating system for accountable AI agents you own. Organizations run more of their operations and services on agents, on any model and any server. Every run is checked against policy, and every agent answers to an owner.
Book a demo to see how to benchmark the models you're considering, in Arabic and English, on your own data.