Robot simulator evaluation: choose for verification, not only for cinema

Robot teams often pick a simulator the way film crews pick a camera: whichever produces the most impressive picture under deadline. That can be rational for data generation and demos. It is a weak strategy for evaluation.

Verification cares about different properties than cinema. If you conflate them, you will trust the wrong stack at the wrong time.

Two jobs, two scorecards

Training / synthesis scorecard

• visual diversity and asset ecosystem,
• throughput of rollouts,
• ease of domain randomization,
• integration with learning frameworks,
• and “does this look good in a talk?”

Evaluation / diligence scorecard

• seed controllability and reproducibility,
• inspectable contact and constraint behavior,
• stable numerics under mass/friction sweeps,
• sensor models you can ablate,
• logging rich enough for failure taxonomies,
• and whether a third party can re-run your protocol.

MuJoCo, Isaac Sim, PyBullet, Gazebo, and cousins each make different tradeoffs across these axes. Haga does not endorse a single simulator as universally “best.” We care whether your chosen stack can support an honest protocol — and whether someone independent checks the physics assumptions sitting inside it.

Cinematic fidelity is not physics trust

Ray-traced reflections do not guarantee that your contact solver tells a coherent story when object mass doubles. A beautiful RGB observation can still sit on brittle dynamics that collapse under friction resampling.

Conversely, a visually modest simulator with deterministic seeds and clear contact events can be an excellent diligence instrument.

Ask:

1. Can I reproduce trial 417 bit-for-bit next quarter?
2. Can I sweep mass and friction without the solver falling apart silently?
3. Can I tell why a trial failed from logs alone?
4. Can an outsider follow the protocol without tribal knowledge?

If the answer is “our demo reel is gorgeous,” you answered a different RFP.

What independent evaluation adds above the sim choice

Even a strong simulator does not solve the self-grading problem. The team that owns assets and reward shaping can still tune the world until the policy passes.

Independent evaluation sits above simulator choice:

• frozen protocols and thresholds,
• adversarial physics tiers,
• shown failures,
• and claim boundaries that keep sim evidence from posing as hardware certification.

Haga’s policy pillar runs adversarial mass/friction stress on robosuite Lift, Stack, PickPlaceCan, and Door with Wilson CIs and severe-tier failure galleries. That methodology is portable in spirit across stacks. See methodology and lab results.

If generative worlds feed your training loop, add world-model physics QA as well — real quiet controls and trajectory checks, not only FVD. Physics-IQ lab shows the shape of that evidence.

A simulator selection checklist for diligence

Use this when the question is “can we evaluate here?” rather than “can we make a trailer?”

Criterion	Why it matters
Seed / determinism controls	Reproducible packets beat folklore
Contact & constraint introspection	Failure taxonomy needs signals
Parameterization of mass, friction, damping	Adversarial tiers require knobs
Sensor ablation (RGB, depth, proprio)	Separates perception luck from control
Logging bandwidth	Diligence is evidence, not averages
Licensing & shareability	Partners may need to inspect setup
Cost at eval scale	Evaluation needs many trials, not one hero run

You may still use a cinematic simulator for marketing and a stricter stack for gates. Dual-stack is allowed. Dual-claim without labels is not.

Common failure patterns we see

1. Training in Sim A, evaluating only on the training distribution of Sim A. Circular.
2. Changing assets mid-campaign without bumping protocol version. Invisible drift.
3. Reporting success without severe tiers. Mild physics flatters everyone.
4. No negative controls for generative augmentations. Synthetic data can poison quietly.
5. Assuming GPU path equality with CPU reference path. Numerics matter.

How to talk about simulators without overclaiming

Good language:

• “Under protocol door-friction-v2 in environment X, success was Y% (Wilson CI …) with failure modes Z.”

Bad language:

• “Our simulator proves the robot is ready.”
• “Photorealism means transfer is solved.”
• “We beat the benchmark” without naming seeds, splits, and non-claims.

The industry already knows the sim-to-real gap is large — Stanford AI Index (2026) style figures like 89.4% sim success versus ~12% real household success exist because measurement and deployment parted ways. Do not widen that gap with prettier lies.

Where Haga fits

Haga is not trying to replace your simulator. We verify policies and world-model outputs with independent, sim-first physics protocols — private packets, not a public leaderboard vanity metric, not a training loop, not Cosmos/NIM cosplay.

Choose simulators deliberately. Then put an evaluation layer on top that can say no.

Request an eval or read the methodology to see how we structure evidence.