HumanEval.

HumanEval is a foundational benchmark created by OpenAI to measure the code generation abilities of large language models on practical programming tasks. The dataset consists of 164 hand-written Python programming problems, each with a function signature, docstring description, and comprehensive unit tests. Models are evaluated using the pass@k metric, which measures the percentage of problems solved when the model is allowed k attempts, providing a standardized way to compare coding performance across different AI systems.

Each HumanEval problem presents a clear programming challenge with natural language descriptions and expected behavior defined through unit tests. The benchmark covers fundamental programming concepts including string manipulation, mathematical operations, data structure handling, and algorithmic problem-solving. The pass@k evaluation methodology accounts for the stochastic nature of language models by allowing multiple generation attempts, with pass@1 measuring success on the first try and pass@10 or pass@100 measuring success when given more attempts.

HumanEval has become the de facto standard for evaluating coding capabilities in AI research and industry, though it has limitations including focus only on Python, relatively simple problems, and potential data contamination as models may have seen similar problems during training. While newer benchmarks like SWE-Bench test more complex real-world scenarios, HumanEval remains valuable for its simplicity, reproducibility, and widespread adoption, making it essential for comparing coding performance across different language models and tracking progress in AI-assisted programming.