MMLU.

MMLU (Massive Multitask Language Understanding) is one of the most widely used benchmarks for evaluating the breadth of knowledge and reasoning capabilities in large language models. Spanning 57 subjects including mathematics, history, computer science, law, and medicine, it tests models from elementary school through professional graduate levels. MMLU has become a standard metric for comparing model capabilities, with scores often cited in research papers and model announcements as evidence of general intelligence.

The benchmark consists of multiple-choice questions with four options each, designed to test both factual knowledge and reasoning across diverse domains. Questions range from basic arithmetic and reading comprehension to advanced topics like organic chemistry and jurisprudence. Models are evaluated in a few-shot setting, typically given 5 examples before answering test questions. The scoring is straightforward: percentage of questions answered correctly, with random guessing yielding 25% accuracy.

While MMLU provides valuable insights into model capabilities, it has limitations that researchers increasingly recognize. The multiple-choice format may not capture nuanced understanding or practical application of knowledge. Some critics argue that high MMLU scores don't necessarily translate to real-world performance, and the benchmark may be susceptible to memorization rather than true comprehension. Despite these concerns, MMLU remains influential in model development and serves as a useful baseline for comparing general knowledge capabilities across different AI systems.