Evidence-first authority page
SWE-bench Leaderboard: What It Measures and How to Read It
What SWE-bench actually measures, its Verified/Lite/Full splits, and how to read leaderboard results without over-generalizing from them.
Direct answer
SWE-bench is a benchmark that evaluates whether a coding agent can resolve real GitHub issues by generating a patch that passes the project's own test suite. This site does not republish a live numeric leaderboard snapshot, since standings change frequently and a stale copy would misrepresent current results — check the benchmark's own maintained leaderboard for current standings.
Last reviewed 2026-08-29
This page remains noindex while its page-level source receipts are completed and verified. It is available for review, not presented as publication-ready authority.
What SWE-bench actually tests
SWE-bench sources real, resolved GitHub issues from popular open-source Python repositories, then asks a system to generate a patch given the issue description and repository state. The patch is scored by whether it makes the project's own associated tests pass, without the system having seen the actual human-authored fix. This makes it a test of real-world issue resolution, not a synthetic coding puzzle.
The dataset splits, and why they matter
- Full — the complete original dataset.
- Lite — a smaller, curated subset designed for faster, cheaper evaluation runs.
- Verified — a human-filtered subset specifically checked for solvability and test correctness, addressing known noise issues in the original dataset.
- A system's score is only comparable to another system's score on the same split — comparing a Lite score to a Verified score is comparing different tests, not the same test at different difficulty.
Why to be cautious generalizing from any leaderboard position
SWE-bench specifically tests Python open-source issue resolution — performance on it doesn't necessarily predict performance on a private codebase, a different language, or a task that isn't a discrete bug-fix (a greenfield feature, a large refactor). Treat a strong SWE-bench result as evidence the system is competent at this specific class of task, and validate on your own representative cases — see how to evaluate AI agents — before generalizing further.