Evaluating Real-World Robot Manipulation Policies in Simulation
SIMPLER is a simulation-based evaluation suite for generalist manipulation policies. It recreates common real-robot setups in simulation (Google Robot and WidowX with the Bridge setup), and its authors report that simulated success correlates with real-world performance and reflects sensitivity to distribution shifts. The environments and the workflow for creating new ones are open-sourced.
How well simulated success predicts a manipulation policy's real-world success, and how sensitive the policy is to visual and environment changes. Results are success rates under two protocols: visual matching and variant aggregation.
Visual matching overlays real camera images onto simulated backgrounds so that the simulated appearance matches the real setup. Variant aggregation creates simulated variants with different backgrounds, lighting, distractors and table textures, and averages the results. The repository provides 10 tasks (6 Google Robot, 4 WidowX) and supports policies such as RT-1, RT-1-X and Octo. The environments are built on SAPIEN and ManiSkill2, and the Bridge environments were also integrated into ManiSkill3.
Real-world evaluation of generalist policies is slow and hard to reproduce, so a simulated proxy that tracks real results gives a repeatable way to rank policies. Its robots are a Google Robot and a WidowX arm, so humanoid relevance is low, although the protocol applies to any policy.
Scoring
| Metric | Key | Unit | Direction |
|---|---|---|---|
| Success rate (visual matching and variant aggregation protocols) | success_rate | Unitless | Higher is better ↑ |
Hardware
No in-dataset robot embodiments are linked.
Simulated Google Robot and WidowX arm (Bridge setup). Neither is a humanoid, so no roster robot is linked.
Recorded outcomes
No result files yet
Evidence
Related evaluations