Four-Factor Integrity Breakdown & Primary-Source Evidence
Privacy
Weight 40%
7.5/10
Access Model: Hybrid Public / Private
"A first-of-its-kind collection, this set includes 276 instances sourced from 18 private, proprietary codebases from startups. These codebases were acquired through partnerships and are not publicly accessible. This set is designed as the ultimate test of generalization on complex, industrial-grade code that is not publicly accessible and is unlikely to have been included in model training data. Results from this challenging dataset are reported on a separate Leaderboard."
"The largest of the three, this private set contains 858 instances. Similar to the public set, it is sourced from a separate group of public repositories with copyleft licenses. This entire dataset is held-out for future analysis and internal evaluations."
Originality
Weight 25%
4/10
Reuse Profile: Medium Reuse (25-75%)
"SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of 41 actively maintained repositories spanning business applications, B2B services, and developer tools. The benchmark is partitioned into a public set with open access to problems sourced from 11 repositories, a held-out set of 12 repositories and a commercial set of 18 proprietary repositories where we have formal partnership agreements with early-stage startups."
Freshness
Weight 20%
10/10
Initial Release: 2025-09Latest Refresh: 2026-09Rolling Updates: No
"(9/22) SWE-Bench Pro V2 released. 642 validated tasks in Harbor format"
Integrity: Independent Offline (Leak-Verified)Sandbox: offline isolated
"V2 images do not contain the fixing commit, so the v1 form of this command (checkout of test files from the fix commit) no longer applies."
Benchmark maintainers can embed this live SVG badge in GitHub READMEs or documentation:
[](https://benchaudit.org/benchmarks/swe-bench-pro/)