Attacked before it ships
Every grader faces degenerate strategies before release. 400 cheating attempts on the Puzzle Benchmark; none got through.
Request the auditVerifiers
Verifiers need verification, too.
Even the measure of intelligence must prove itself.
Why evaluation needs
to be more reliable
59.4%
Of 138 difficult tasks reviewed by OpenAI, 59.4% had substantial test or task-description flaws. On 23 February 2026, OpenAI announced it would stop reporting SWE-bench Verified results. ↗
296
METR had four maintainers of scikit-learn, Sphinx and pytest review 296 AI-generated pull requests that had already passed the automated grader. Published 10 March 2026: maintainer merge rates ran 24.2 percentage points below the grader score. ↗
30%
On 8 July 2026, OpenAI audited SWE-Bench Pro (published by Scale AI), the benchmark it had recommended five months earlier as the replacement for SWE-bench Verified, and estimated that about 30% of its 731 public tasks are broken. An automated pipeline flagged 200; a parallel human annotation campaign flagged 249. OpenAI retracted the recommendation. ↗
Match the final state
Compare the final database with an annotated target state. An agent may reach that state through incorrect actions and still pass; each task also needs its own annotation. The approach below closes this gap.
Check process invariants
Encode business rules that must hold throughout the workflow. Tasks in a domain share these rules, and each step must cite its evidence.
How we keep scores honest
A score is only as good as the grader behind it.
Where every score comes from
Five sources of truth. None of them is a model.
- Competition platformsEach task is a past machine-learning competition. Its held-out labels never leave the platform that ran it, and that platform scores the submission.SimReal-MLBench
- Exact answersHard, human-written puzzles, held out so they measure reasoning, not recall. Every answer is checked exactly; the reward is binary.Puzzle Benchmark
- Mathematical proofA working solution, or a certificate that none exists. No proof, no score.MathmoBench
- Real eventsForecasts are scored when the event happens, against a baseline sealed in advance.Future Prediction Bench
- Simulated futuresA frozen patch is run against the codebase’s next releases.SWE-Forward