HUMAIN puts ALLaM through an independent human evaluation, in Arabic and English

HUMAIN builds ALLaM, an Arabic large language model. Before an organization trusts a model with its work, it wants to know where that model stands: against which models, on which tasks, and by whose judgment. In late 2025, aixplain ran an independent human evaluation of ALLaM for HUMAIN, and extended the annotation app its team uses to judge model output.

The challenge: scores miss what people notice

Automatic metrics show how close an answer is to a reference. They say little about whether an Arabic answer reads naturally, follows the instruction, stays safe or gets the facts right. For that, people have to judge.

Those judgments had to be fair: the same prompts for every model, answers judged without knowing which model wrote them, and reviewers who agree with each other often enough for the result to hold.

And the results had to be usable. HUMAIN's model team needed to see where ALLaM wins, where it loses, and why.

Why aixplain: evaluation built in Arabic

• Arabic first. aixplain's language team writes prompts and reviews answers in Modern Standard Arabic and English, with native speakers judging the Arabic.
• Any model, one interface. Every model's answers are collected the same way, so the comparison is about the answers and nothing else.
• A method, end to end. Prompt design, blind review, rubrics, issue tags and quality checks, run on an annotation platform aixplain has used for Arabic evaluation before.

The solution: a blind, pairwise human evaluation

What was tested. Prompts covered the capabilities buyers ask about:

• Robustness: hallucination, consistency, language mixing and long contexts.
• Steerability: following system prompts and complex instructions, keeping answers short when asked, structured output in JSON and tables, and translation.
• Knowledge and reasoning: logic, math, coding including SQL, and general knowledge.
• Safety and alignment: resisting jailbreaks and prompt injection, cultural sensitivity and refusing harmful requests.
• Agentic work: answering with web search, and producing structured reports with tables and plots.

How it was judged. Curated prompts in Arabic and English went to ALLaM and to leading open and closed models. The answers were anonymized and shuffled. Bilingual reviewers then compared them in pairs: a rubric score for each answer, tags for failures such as hallucination, inconsistency, verbosity or formatting, a preference and how strong it was, the main reason for it, and a short justification.

How it stayed fair. Language quality checks ran throughout, and agreement between reviewers was tracked against a target set before the work began.

What HUMAIN received. A report with model rankings, win and loss rates, reviewer agreement and qualitative findings; the full annotated dataset; and recommendations for refining ALLaM.

The app. aixplain also extended Tawseem, the Label Studio-based annotation app the team uses, with what HUMAIN asked for: pairwise comparison data and progress tracking for annotators.

Benchmarking: how you choose your own model

The aixplain OS runs on any model, including the one you build or host. That freedom pays off only if you can tell which model is right for the work. Benchmarking is how aixplain answers that, before an agent goes live.

• Your data. A benchmark runs on your own prompts, documents or recordings, or on public datasets.
• Automatic metrics. For text, ROUGE and BLEU against reference answers. For speech, word, character and match error rates (WER, CER, MER) and word information lost (WIL). Text can be normalized first, so a capital letter or a number written as a word doesn't count as an error.
• Human evaluation. Rubric scores, pairwise preferences, issue tags and reviewer agreement, for what metrics can't see.
• Cost and speed. Cost per call, latency and the share of failed calls for every model, next to its quality.

Evaluator, coming soon, brings the same scoring to agents: it scores an agent's runs over time, so a new model or a new version has to show it does better.

What's next: every new version, the same test

Each new ALLaM version can go through the same evaluation, on the same prompts and rubric, so changes show up as gains or losses against the version before. When ALLaM runs inside agents on the aixplain OS, every agent answers to an owner, inspector agents check every run, and every run is on record.

See aixplain at work

aixplain is the operating system for accountable AI agents you own. Organizations run more of their operations and services on agents, on any model and any server. Every run is checked against policy, and every agent answers to an owner.

Book a demo to see how to benchmark the models you're considering, on your own data.