How well do today's LLMs understand African health realities?
AfriMedEval is the evaluation workspace for AfriMed-QA — a pan-African, multi-specialty medical question-answering benchmark built with clinicians across the continent. We measure correctness, localization gaps, and clinical usefulness where global medical benchmarks fall short.
15,275+
Medical questions
600+
Clinician contributors
32
Medical specialties
16
African countries
Why this gap matters
Strong on MedQA does not mean ready for African care.
Large models look capable on Western medical exams, yet AfriMed-QA reveals uneven generalization across African specialties, geographies, and patient contexts — exactly the settings where safe deployment matters most.
Western benchmarks fall short
USMLE, MedQA, and PubMedQA miss linguistic variation, local disease burden, and region-specific clinical practice.
Uneven specialty performance
Top models can look strong overall, yet underperform in OBGYN, pediatrics, infectious disease, and surgery.
Country-level gaps
AfriMed-QA found clear geographic variation — models that pass global tests still miss African contexts.
Built across the continent
Real clinicians. Real African contexts.
Questions and evaluations come from medical professionals working in African health systems — capturing the diseases, languages, and realities that Western datasets overlook.
621
Contributors
60+
Medical schools
16
Countries
The AfriMed-QA benchmark
A pan-African medical QA standard
Sourced from 60+ medical schools across 16 countries, with multi-layer quality control and blind human evaluation. Built to stress-test LLMs on Africa-centric clinical knowledge, not just exam regurgitation.
Multiple Choice
Expert- and trainee-authored exam-style questions spanning 32 specialties, grounded in African clinical curricula.
Short Answer
Open-ended clinical reasoning prompts that test recall, differential thinking, and contextual judgment.
Consumer Queries
Patient-facing health questions that reveal how models respond to everyday African healthcare concerns.
Built with clinicians
621 contributors. Expert review. Blind evaluation.
Questions and judgments come from African medical professionals, not synthetic Western proxies. Multi-layer QC keeps the benchmark trustworthy for research and deployment decisions.
What evaluation revealed
- Models often score higher on MedQA than on AfriMed-QA expert MCQs
- Large general models can outperform smaller biomedical specialists
- Clear underperformance on several high-stakes African specialties
- Meaningful accuracy variation by country of question origin
The AfriMedEval method
From benchmark to clinical judgment
Curate the benchmark
AfriMed-QA questions across 32 specialties and 16 countries, quality-controlled by African clinicians.
Run the models
Generate and judge responses with reproducible LLM-as-a-Judge pipelines on MCQ, SAQ, and consumer queries.
Review with clinicians
Blind human evaluation for correctness, safety, localization, and clinical usefulness, plus translation review.
Measure the gap
Compare scores by model, specialty, country, and language to surface where deployment is safe.
The platform
One workspace for the whole evaluation
AfriMedEval turns the AfriMed-QA research foundation into an operational evaluation studio — human review, translation assessment, and LLM judging in one place.
Human clinical evaluation
Invite clinicians to rate model outputs for correctness, safety, cultural fit, and clinical usefulness.
Machine translation review
Evaluate medical translations across African languages with structured metrics and side-by-side review.
LLM-as-a-Judge runs
Benchmark generator and judge models on AfriMed-QA style datasets with reproducible scoring pipelines.
Results & export
Track evaluation history, compare scores across models and languages, and export results for research.
AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset
Presented at ACL 2025 and at FEF-AI4SD 2026. The paper benchmarks 30 LLMs and shows where medical models are promising — and where human supervision remains essential.
Looking forward
- Expanded dataset coverage across more specialties and locales
- Multilingual and multimodal extensions for real African clinical settings
- Comprehensive human + automated evaluation through AfriMedEval
Evaluate models for African healthcare.
Sign in to run human evaluations, review medical translations, or continue AfriMed-QA style benchmarking with your team.