SWE-Bench.

A benchmark that measures an AI agent's ability to resolve real GitHub issues from open-source repositories — a rigorous test of practical software engineering capability.

SWE-Bench (Software Engineering Benchmark) was introduced by Princeton NLP in 2024. It consists of 2,294 real GitHub issues from popular Python open-source repositories like Django, scikit-learn, and Flask. Each issue comes with a codebase snapshot, the issue description, and a test suite that validates the correct fix. An AI agent must understand the issue, locate the relevant code, implement a fix, and pass the tests.

SWE-Bench is widely considered the most practically meaningful benchmark for coding AI because it tests the full software engineering workflow — not just generating code from scratch, but understanding existing codebases, reproducing bugs, and writing targeted fixes. Performance on SWE-Bench correlates more closely with real-world engineering utility than benchmarks like HumanEval.

The SWE-Bench Verified subset (500 human-validated problems) is used most often for direct model comparisons. Scores have improved dramatically: early agents scored under 5%, while frontier agents in 2025 exceed 50% on the verified set. Aider, Devin, and Claude Code are among the tools evaluated on SWE-Bench.