Domains: Coding
Skills: CodingAgenticTool UseReasoningLong Context

Four-Factor Integrity Breakdown & Primary-Source Evidence

Privacy

Weight 40%
7.5/10
Access Model: Hybrid Public / Private
"A first-of-its-kind collection, this set includes 276 instances sourced from 18 private, proprietary codebases from startups. These codebases were acquired through partnerships and are not publicly accessible. This set is designed as the ultimate test of generalization on complex, industrial-grade code that is not publicly accessible and is unlikely to have been included in model training data. Results from this challenging dataset are reported on a separate Leaderboard."
"The largest of the three, this private set contains 858 instances. Similar to the public set, it is sourced from a separate group of public repositories with copyleft licenses. This entire dataset is held-out for future analysis and internal evaluations."

Originality

Weight 25%
4/10
Reuse Profile: Medium Reuse (25-75%)
"SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of 41 actively maintained repositories spanning business applications, B2B services, and developer tools. The benchmark is partitioned into a public set with open access to problems sourced from 11 repositories, a held-out set of 12 repositories and a commercial set of 18 proprietary repositories where we have formal partnership agreements with early-stage startups."
Primary source (Paper) ↗ Verified 2026-11-17

Freshness

Weight 20%
10/10
Initial Release: 2025-09Latest Refresh: 2026-09Rolling Updates: No
"(9/22) SWE-Bench Pro V2 released. 642 validated tasks in Harbor format"
Primary source (GitHub) ↗ Verified 2026-11-17

Grading

Weight 15%
10/10
Integrity: Independent Offline (Leak-Verified)Sandbox: offline isolated
"V2 images do not contain the fixing commit, so the v1 form of this command (checkout of test files from the fix commit) no longer applies."
Primary source (HuggingFace) ↗ Verified 2026-11-17

Embeddable Audit Badge

View Markdown (.md)

Benchmark maintainers can embed this live SVG badge in GitHub READMEs or documentation:

BenchAudit SWE-bench Pro Tier A 7.5/10
[![BenchAudit Tier A](https://benchaudit.org/badges/swe-bench-pro.svg)](https://benchaudit.org/benchmarks/swe-bench-pro/)