Benchmark.
A standardized test or set of tasks used to evaluate and compare the capabilities of different AI models on a common scale.
AI benchmarks are curated datasets and evaluation protocols designed to measure specific capabilities of language models. They allow researchers, developers, and the public to make meaningful comparisons between models from different companies and generations. Common benchmarks include MMLU (broad knowledge), HumanEval (coding), and GPQA (PhD-level reasoning).
A good benchmark should be challenging enough to differentiate strong from weak models, diverse enough to cover a range of real-world tasks, and resistant to 'contamination' — the risk that test questions appear in training data. Benchmark contamination is a real concern, as models trained on web-scale data may have seen benchmark questions during training.
No single benchmark tells the full story. Real-world performance often diverges from benchmark scores, which is why the community relies on a diverse ecosystem of evaluations. Human preference rankings on platforms like LMSYS Chatbot Arena complement automated benchmarks by capturing qualities like helpfulness and naturalness that are hard to quantify.
Related terms.
- MMLUA comprehensive benchmark evaluating language models across 57 academic subjects from elementary to professional level using multiple-choice questions.
- HumanEvalA benchmark dataset of 164 Python programming problems used to evaluate code generation capabilities of language models through unit test pass rates.
- GPQAA graduate-level, 'Google-proof' multiple-choice benchmark of hard science questions designed to resist easy lookup and test deep reasoning.
- Chatbot ArenaA public platform where users anonymously compare language models in head-to-head battles, with results aggregated into Elo ratings to create crowdsourced leaderboards.