# SWE-bench Pro Contamination Audit

> SWE-bench Pro scores 7.5/10 (Tier A, Low Risk) on BenchAudit as of November 2026, reflecting a hybrid public and private held-out split and isolated offline third-party grading.

- **Canonical URL:** https://benchaudit.org/benchmarks/swe-bench-pro/
- **Benchmark ID:** `swe-bench-pro`
- **Hub:** Scale AI
- **Category:** Coding
- **Domains:** Coding
- **Skills:** Coding, Agentic, Tool Use, Reasoning, Long Context
- **BenchAudit Score:** 7.5/10 (Tier A - Low Risk)
- **Last Audited:** 2026-11-17 (Ref 2026-11)

## Factor Scores

| Factor | Score (/10) | Weight |
| --- | ---: | ---: |
| Privacy | 7.5 | 40% |
| Originality | 4 | 25% |
| Freshness | 10 | 20% |
| Grading | 10 | 15% |

## Primary-Source Evidence

### Privacy
> "A first-of-its-kind collection, this set includes 276 instances sourced from 18 private, proprietary codebases from startups. These codebases were acquired through partnerships and are not publicly accessible. This set is designed as the ultimate test of generalization on complex, industrial-grade code that is not publicly accessible and is unlikely to have been included in model training data. Results from this challenging dataset are reported on a separate Leaderboard."
- Source: https://labs.scale.com/leaderboard/swe_bench_pro

### Originality
> "SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of 41 actively maintained repositories spanning business applications, B2B services, and developer tools. The benchmark is partitioned into a public set with open access to problems sourced from 11 repositories, a held-out set of 12 repositories and a commercial set of 18 proprietary repositories where we have formal partnership agreements with early-stage startups."
- Source: https://arxiv.org/abs/2509.16941

### Freshness
> "(9/22) SWE-Bench Pro V2 released. 642 validated tasks in Harbor format"
- Source: https://github.com/scaleapi/SWE-bench_Pro-os

### Grading
> "V2 images do not contain the fixing commit, so the v1 form of this command (checkout of test files from the fix commit) no longer applies."
- Source: https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro
