Four-Factor Integrity Breakdown & Primary-Source Evidence
Access Model: Public Plaintext
"A human-validated subset of 500 SWE-bench instances for reliable evaluation of coding agents and language models."
Originality
Weight 25%
4/10
Reuse Profile: Medium Reuse (25-75%)
"To this end, we introduce SWE-bench, an evaluation framework consisting of $2,294$ software engineering problems drawn from real GitHub issues and corresponding pull requests across $12$ popular Python repositories."
Freshness
Weight 20%
0/10
Initial Release: 2024-08Latest Refresh: 2024-08Rolling Updates: No
"**[Aug. 13, 2024]**: Introducing *SWE-bench Verified*! Part 2 of our collaboration with [OpenAI Preparedness](https://openai.com/preparedness/). A subset of 500 problems that real software engineers have confirmed are solvable."
Grading
Weight 15%
3.3/10
Integrity: Self-Reported / Live NetworkSandbox: offline isolated
"Check out the main SWE-bench repository docs for instructions on how to generate and evaluate predictions on SWE-bench [Lite, Verified, Multimodal]."
Benchmark maintainers can embed this live SVG badge in GitHub READMEs or documentation:
[](https://benchaudit.org/benchmarks/swe-bench-verified/)