01

Ground rules

You cannot inspect closed model weights to prove a public test set stayed out of training. We evaluate only what can be verified from primary sources.

01.1

Public files enter training runs

If questions and answers sit openly on GitHub or Hugging Face, automated crawlers and synthetic pipelines will eventually pick them up. Canary strings, non-redistribution notices, and honor-system pledges offer no real protection.

01.2

Click-through gates do not hold

Forms that auto-approve downloads or publish password-locked archives next to their decryption code are easy to automate around. Only manual identity verification or held-out private splits keep items restricted.

01.3

Every label needs a source

Each classification cites an official paper, code repository, dataset card, or documentation page with a direct HTTPS link and a verbatim quote. Unverified safeguards are never assumed.

01.4

Contamination scope only

We measure whether a benchmark lets models memorize answers or cheat the runner. We do not rate task difficulty, domain coverage, or LLM-as-a-judge bias unless test verifiers leak.

02

What we evaluate

Four structural criteria determine a benchmark’s contamination-resistance rating from verified facts.

Four Factors: Exposure · Provenance · Freshness · Harness

Every factor and the Overall column appear on a 0โ€“10 scale on the index for direct comparison.

Exposure Access to prompts, answers, and graders Provenance Original authoring vs. reused sources Freshness Exposure age and active leak pruning Harness Sandbox isolation and leak audits

Exposure & Access

Who can download the test prompts, reference answers, and grading scripts?

Access level What it means
Strictly Private Held-Out Prompts, environments, and graders stay behind a private server with zero public downloads.
Hybrid Private + Public Publishes a sample or dev split while headline evaluations run on a private held-out set.
Public Prompts, Private Answers Questions are public, while ground-truth answers and grading run on an official server.
Identity-Vetted (KYC) Downloads require manual researcher vetting and signed access agreements.
Auto-Gated / Encrypted Archive Uses an auto-approved form gate or ships encrypted files with decryption keys in public code.
Public Plaintext Full questions and reference answers sit openly on a website, repository, or dataset hub.

Source Provenance

How much of the benchmark was written from scratch instead of reused from existing public sets?

Pre-existing share What it means
All Original Written from scratch specifically for this benchmark, with no prior public footprint.
Low Reuse Mostly original tasks built on real workflows or collected continuously after model cutoffs.
Medium Reuse Mix of newly written items, curated public tasks, or rewritten multiple-choice variants.
High Reuse Drawn mostly from older exams, coding contests, textbooks, Wikipedia, or earlier datasets.

Freshness

How long have the current tasks been exposed relative to advancing training cutoffs?

Age is measured in months from the latest substantive update to the current audit window.

Freshness profile What it means
Strictly Private or Rolling Post-Cutoff Private held-out suites and rolling post-cutoff pipelines retain integrity even as months pass.
Hybrid Split or KYC Vetted Private splits hold up over time, while public samples or vetted copies still carry gradual exposure risk.
Public Prompts, Private Grader Visible questions age steadily as new web crawls capture public prompt text.
Public Static or Auto-Gated Open files and click-through downloads lose integrity rapidly as training pipelines ingest public copies.
Versioned Leak Pruning Maintained releases that actively retire leaked, saturated, or broken items preserve higher freshness than frozen snapshots.

Runtime Harness

Can a model or coding agent leak answers during execution via network access, container files, or git history?

Execution setup What it means
Offline Sandbox + Leak Audits Run independently in network-disabled containers with stripped git history, isolated graders, and trajectory audits.
Independent Clean Harness Evaluated by a third party using standardized prompts and clean extraction, with no documented leaks.
Self-Reported or Live Internet Relies on provider self-reporting or lets agents browse the live web while solving tasks.
Known Container or Git Leak Documented runner bugs allow models to inspect commit logs, future branches, or local test suites.
03

Risk tiers (Sโ€“D)

Overall ratings map to five contamination-risk tiers.

Tier Score Rating What it means
Tier S 85โ€“100 Fortified Private or continuously updated post-cutoff tasks, original authoring, and isolated execution. Hardest to game in training or at runtime.
Tier A 70โ€“84 Low Risk Protected by held-out test splits, blind grading servers, identity vetting, or fresh original tasks with minimal reuse.
Tier B 50โ€“69 Moderate Risk Public or lightly gated tasks partially offset by new authoring, recent refreshes, or clean third-party harnesses.
Tier C 30โ€“49 High Risk Public or auto-gated datasets paired with older release dates or recycled exam material. Training overlap is likely.
Tier D 0โ€“29 Compromised Long-exposed public files or recycled corpora. Treat high scores here as saturated or likely memorized.
04

Evidence & updates

Every classification cites primary sources and updates from verified fact records.

Inspectable quotes on every benchmark

Click any row on the index to open its audit drawer. Each factor lists the direct primary-source URL and exact quoted text behind that rating.

Maintainer claims vs. technical enforcement

When authors publish canary tokens, non-redistribution rules, or contamination notes, the drawer quotes those guarantees side by side with how well they hold up in practice.

Audit window and leaderboard sync

Structural audits evaluate against the 2026-11 reference date and update from verified fact records without manual overrides. Model standings in the Insights section track official leaderboards.