Pass@k.

Pass@k is a fundamental evaluation metric for code generation models that measures the likelihood of producing at least one correct solution when sampling k different attempts. Unlike simple accuracy metrics that only consider the first generated solution, Pass@k accounts for the stochastic nature of language models and their ability to produce multiple valid approaches to the same problem. This metric is crucial for understanding how reliably a model can solve coding tasks when given multiple attempts, which better reflects real-world usage patterns where developers often iterate on solutions.

The metric works by generating k independent code samples for a given programming problem, then checking if any of these samples pass all provided test cases. Common values include Pass@1 (success rate on first try), Pass@10, and Pass@100, with higher k values generally showing better performance as they allow more opportunities for success. The calculation accounts for the combinatorial probability that at least one solution succeeds, making it more forgiving than strict first-attempt accuracy while still maintaining meaningful standards for code quality and correctness.

Pass@k has become the standard evaluation method for coding benchmarks like HumanEval and is essential for comparing different models' coding capabilities. However, it has limitations: it doesn't measure code quality, efficiency, or maintainability, only functional correctness. The metric can also be influenced by test suite completeness and may not capture edge cases or real-world complexity. Despite these constraints, Pass@k remains invaluable for assessing whether AI coding assistants can reliably produce working solutions, helping developers understand when and how to trust automated code generation.