Domains: Coding
Skills: CodingAgenticTool UseReasoningLong Context

Four-Factor Integrity Breakdown & Primary-Source Evidence

Privacy

Weight 40%
0/10
Access Model: Public Plaintext
"We're releasing SWE-Bench Pro V2, a refreshed public split with a modified benchmark and a locked evaluation protocol, co-developed with Reflection."
"The same data is on HuggingFace as ScaleAI/SWE-bench_Pro (config default = V2, hard = HARD-51, v1 = the original 731 tasks)."

Originality

Weight 25%
4/10
Reuse Profile: Medium Reuse (25-75%)
"Instructions: 529 problem statements rewritten so that every graded assertion traces back to a sentence in the text (only problem statement, requirements and interfaces changed); each correction was implemented blind by a second engineer and graded by the task's own hidden tests."
Primary source (GitHub) ↗ Verified 2026-11-20

Freshness

Weight 20%
7/10
Initial Release: 2026-09Latest Refresh: 2026-09Rolling Updates: No
"V2 (2026-09-22) is the default config: 642 tasks (go 256, python 237, js 145, ts 4)."
Primary source (HuggingFace) ↗ Verified 2026-11-20

Grading

Weight 15%
10/10
Integrity: Independent Offline (Leak-Verified)Sandbox: offline isolated
"Every agent diff is re-graded on a pristine image, and we publish both grades."

Frontier Model Standings on SWE-bench Pro V2

Top 12 active models tracked
Rank Model Official Score Percentile
#1Gemini 4 Argon high Google99.2%92.5p
#2Claude Fable 5.1 max Anthropic99.1%90.9p
#2Inkling 2 Small default Thinking Machines97.7%82.9p
#3Muse Wonder max Meta96.7%73.3p
#4GPT-6 Astra max OpenAI96.9%72.7p
#4MiniMax-M3.1-Flash max MiniMax96.6%63.6p
#5GPT-6.1 Sol max OpenAI96.1%54.0p
#6Claude Sonnet 5.5 max Anthropic95.8%44.4p
#7Muse Spark Open xhigh Meta95.0%34.8p
#8Claude Haiku 5.5 max Anthropic87.2%25.1p
#9GLM-5.3 max Z AI95.6%27.3p
#11Gemini 3.8 Flash high Google94.9%9.1p

Embeddable Audit Badge

View Markdown (.md)

Benchmark maintainers can embed this live SVG badge in GitHub READMEs or documentation:

BenchAudit SWE-bench Pro V2 Tier C 3.9/10
[![BenchAudit Tier C](https://benchaudit.org/badges/swe-bench-pro-v2.svg)](https://benchaudit.org/benchmarks/swe-bench-pro-v2/)