Attacked before it ships
Every grader faces degenerate strategies before release. 400 cheating attempts on the Puzzle Benchmark; none got through.
Request the auditBenchmarks
AI scores you can’t argue with.
Measure intelligence against the people who make the world work.
- 5
- benchmarks, one engine
- 0
- AI judges in any score
- 0 / 400
- cheating attempts that got through
Where every score comes from
Five sources of truth. None of them is a model.
Competition platforms
Each task is a past machine-learning competition. Its held-out labels never leave the platform that ran it, and that platform scores the submission.
SimReal-MLBenchDetailsExact answers
Hard, human-written puzzles, held out so they measure reasoning, not recall. Every answer is checked exactly; the reward is binary.
Puzzle BenchmarkDetailsMathematical proof
A working solution, or a certificate that none exists. No proof, no score.
MathmoBenchDetailsReal events
Forecasts are scored when the event happens, against a baseline sealed in advance.
Future Prediction BenchDetailsSimulated futures
A frozen patch is run against the codebase’s next releases.
SWE-ForwardDetailsHow we keep scores honest
A score is only as good as the grader behind it.
Task tracks
Four tracks, so the test never goes stale.
Public
The released split: tasks anyone with access can inspect, run and reproduce.
Hidden
Withheld tasks that decide the score.
Live
Questions whose answers do not exist yet.
Retired
Released in full once a test can no longer be kept secret.