Four Factors: Exposure · Provenance · Freshness · Harness
Every factor and the Overall column appear on a 0โ10 scale on the index for direct comparison.
Exposure Access to prompts, answers, and graders
Provenance Original authoring vs. reused sources
Freshness Exposure age and active leak pruning
Harness Sandbox isolation and leak audits
Who can download the test prompts, reference answers, and grading scripts?
| Access level |
What it means |
| Strictly Private Held-Out |
Prompts, environments, and graders stay behind a private server with zero public downloads. |
| Hybrid Private + Public |
Publishes a sample or dev split while headline evaluations run on a private held-out set. |
| Public Prompts, Private Answers |
Questions are public, while ground-truth answers and grading run on an official server. |
| Identity-Vetted (KYC) |
Downloads require manual researcher vetting and signed access agreements. |
| Auto-Gated / Encrypted Archive |
Uses an auto-approved form gate or ships encrypted files with decryption keys in public code. |
| Public Plaintext |
Full questions and reference answers sit openly on a website, repository, or dataset hub. |
How much of the benchmark was written from scratch instead of reused from existing public sets?
| Pre-existing share |
What it means |
| All Original |
Written from scratch specifically for this benchmark, with no prior public footprint. |
| Low Reuse |
Mostly original tasks built on real workflows or collected continuously after model cutoffs. |
| Medium Reuse |
Mix of newly written items, curated public tasks, or rewritten multiple-choice variants. |
| High Reuse |
Drawn mostly from older exams, coding contests, textbooks, Wikipedia, or earlier datasets. |
How long have the current tasks been exposed relative to advancing training cutoffs?
Age is measured in months from the latest substantive update to the current audit window.
| Freshness profile |
What it means |
| Strictly Private or Rolling Post-Cutoff |
Private held-out suites and rolling post-cutoff pipelines retain integrity even as months pass. |
| Hybrid Split or KYC Vetted |
Private splits hold up over time, while public samples or vetted copies still carry gradual exposure risk. |
| Public Prompts, Private Grader |
Visible questions age steadily as new web crawls capture public prompt text. |
| Public Static or Auto-Gated |
Open files and click-through downloads lose integrity rapidly as training pipelines ingest public copies. |
| Versioned Leak Pruning |
Maintained releases that actively retire leaked, saturated, or broken items preserve higher freshness than frozen snapshots. |
Can a model or coding agent leak answers during execution via network access, container files, or git history?
| Execution setup |
What it means |
| Offline Sandbox + Leak Audits |
Run independently in network-disabled containers with stripped git history, isolated graders, and trajectory audits. |
| Independent Clean Harness |
Evaluated by a third party using standardized prompts and clean extraction, with no documented leaks. |
| Self-Reported or Live Internet |
Relies on provider self-reporting or lets agents browse the live web while solving tasks. |
| Known Container or Git Leak |
Documented runner bugs allow models to inspect commit logs, future branches, or local test suites. |