Benchmark · AI research
SimReal-MLBench
Tests whether AI can do end-to-end machine-learning work.
- What it measures
- Real machine-learning research across 7 modalities
- Scored by
- Held-out answers kept by competition platforms
- Headline figure
- 60 tasks
- Domain
- AI research
- Applies to
- Automated AI research
- Access
- On request
How it is scored
Competition platforms
Each task is a past machine-learning competition. Its held-out labels never leave the platform that ran it, and that platform scores the submission.
We don’t grade our own exam. The competition platform does. Sixty real machine-learning competitions across seven data modalities; test labels never leave the platform, and scores compare against the human teams who entered. Score is 100 × p², where p is the share of historical entrants beaten.
What it tests
Real ML competitions
- 60 past competitions across 7 data types: tables, vision, language, audio, multimodal, forecasting and science.
- Scored by each competition’s own held-out labels, which never leave the platform.
- Built for labs measuring AI that does machine-learning work.
Experts behind it
Practitioners define what correct means for this benchmark. Paid, remote roles.
Other benchmarks