Benchmark · Reasoning
Puzzle Benchmark
709 puzzles. 400 cheating attempts. None got through.
- What it measures
- Reasoning on 709 hard, human-written puzzles
- Scored by
- Held-out answers, checked exactly
- Headline figure
- 0 / 400 cheats
- Domain
- Reasoning
- Applies to
- Frontier reasoning evaluation
- Access
- On request
How it is scored
Exact answers
Hard, human-written puzzles, held out so they measure reasoning, not recall. Every answer is checked exactly; the reward is binary.
What it tests
Attack-tested reasoning tests
- 709 hard puzzles written by people.
- Hard, human-written and held out, so it measures reasoning, not recall.
- Binary reward, computed locally in microseconds. No judge model and no cost per rollout.
Experts behind it
Practitioners define what correct means for this benchmark. Paid, remote roles.
Other benchmarks