SIMREAL
Passing tests
≠ Doing the work.
We build environments where agents do real work, and use invariants to verify that they did it right.
Why evaluation needs
to be more reliable
Of 138 difficult tasks reviewed by OpenAI, 59.4% had substantial test or task-description flaws. On February 23, 2026, OpenAI announced it would stop reporting SWE-bench Verified results. ↗
METR asked four maintainers of scikit-learn, Sphinx, and pytest to review 296 AI submissions that passed automated grading. Human review and automated scores still diverge. ↗
Gemini 3 Pro accuracy listed on the HLE website. The researchers caution that high scores on closed academic questions alone do not establish autonomous research ability. ↗
The difference verification makes
Both books balance. Only one agent did the work.
Check only whether the trial balance is zero, and both agents pass.
Both checks pass. The close is genuinely complete.
Models already score highly on benchmarks.
For enterprises, the question is whether an agent
will get the work right when nobody is watching.
That is what we test.
Simulation environments
Stateful work scenarios. Snapshot, reset, and practice repeatedly under the same rules.
Verifiers
Business rules expressed as constraints that must always hold, beyond an annotated final answer.
Adversarial testing
Attack our verifiers with degenerate strategies. Publish the results and the checks added in response.
Expert trajectories
Let practitioners do the work. Preserve their actions, reasoning, corrections, and reviews.