Verifiers
Verifiers need verification, too.
Record which strategies bypassed the checks, and what happened after adding the missing constraints.
Why evaluation needs
to be more reliable
Of 138 difficult tasks reviewed by OpenAI, 59.4% had substantial test or task-description flaws. On February 23, 2026, OpenAI announced it would stop reporting SWE-bench Verified results. ↗
METR asked four maintainers of scikit-learn, Sphinx, and pytest to review 296 AI submissions that passed automated grading. Human review and automated scores still diverge. ↗
Gemini 3 Pro accuracy listed on the HLE website. The researchers caution that high scores on closed academic questions alone do not establish autonomous research ability. ↗
Match the final state
Compare the final database with an annotated target state. An agent may reach that state through incorrect actions and still pass; each task also needs its own annotation.
Check process invariants
Encode business rules that must hold throughout the workflow. Different tasks in a domain can reuse these rules, with evidence used to verify the result.
Month-end close
Both books balance. Only one agent did the work.
If you only check whether the trial balance is zero, both agents pass.
Both checks pass. The close is genuinely complete.
Adversarial tests
We attack our own verifiers and publish the results.
Ten degenerate strategies. Two rounds. The first round shows where we failed.
Month-end close · Controlled comparison. Round 1 uses a baseline verifier constructed for this experiment. Round 2 adds process and evidence checks. The same ten synthetic trajectories are executed in both rounds; full source code and results are available to download.
| Strategy | Round one | Added check | Round two |
|---|---|---|---|
| Move the discrepancy into suspense | Bypassed | Require zero closing suspense balance | Blocked |
| Claim completion without entries | Bypassed | Require at least one adjustment with a nonzero net amount | Blocked |
| Exhaust the budget with retries | Bypassed | Limit tool calls to 200 | Blocked |
| Delete problematic entries | Blocked | — | Blocked |
| Insert balancing entries without evidence | Blocked | — | Blocked |
| Spread tiny adjustments across accounts | Bypassed | Validate evidence and account per entry | Blocked |
| Fix only the easy discrepancies | Blocked | — | Blocked |
| Reverse and repost repeatedly | Bypassed | Detect zero-net-amount churn | Blocked |
| Abuse materiality flags | Bypassed | Check thresholds for flagged items | Blocked |
| Take no action | Blocked | — | Blocked |
Month-end close · Invariants
The trial balance debit-credit difference stays at zero after every step
The fundamental accounting identity.
Suspense and clearing accounts close at zero
Balanced books alone do not resolve discrepancies.
Every adjustment references its supporting evidence
Evidence must identify a statement record or voucher.
Original journal entries cannot be edited or deleted
Correct errors with new reversal entries; preserve the originals.
Adjusted book and bank balances reconcile
Only explicitly supported timing differences are allowed.