Sim-to-real benchmark design: what to measure when hardware is scarce

Hardware scarcity is the default condition of robotics. GPUs are easier to book than robot cells; robot cells are easier to book than safe pilot sites. So the field invents sim-to-real benchmarks — and too often invents leaderboards of successes without the protocol discipline that makes transfer claims meaningful.

This post is a design guide for benchmarks you can defend when someone asks, “what exactly did you measure?”

What a credible sim-to-real benchmark includes

1. A frozen protocol, not a vibe

Name the version. Pin seeds, world generators, preprocessing, and scoring scripts. If two labs cannot re-run the same trial definition, you built a club, not a benchmark.

2. Shared world variations

A single map or tabletop is a demo harness. A declared distribution of variations — mass, friction, lighting, clutter, sensor noise — is a benchmark substrate.

3. Separation of sim score from hardware validation score

Never average them into one “sim2real number” without labels. Report:

• S: protocol performance in simulation,
• H: performance on the paired hardware protocol (when available),
• Δ: the gap, with the same failure taxonomy on both sides when possible.

4. An explicit failure taxonomy

Success-only tables hide the science. Slip vs timeout vs localization loss vs contact explosion are different bugs. Benchmarks should force those labels.

5. Uncertainty that travels with the mean

Small n is normal early. Wilson (or similarly honest) intervals keep people from narrating noise as progress.

6. Claim boundaries in the README

State what the benchmark cannot certify: safety cases, arbitrary embodiments, open-world household competence, etc.

Leaderboards without protocols are entertainment

Public rankings can accelerate research when the rules are crisp. They can also create Goodhart pressure: optimize the visible metric, silently specialize to the eval distribution, and ship vocabulary that sounds like deployment readiness.

Haga deliberately is not a public leaderboard product. Our posture is private verification packets with public methodology — useful when companies need diligence without turning their checkpoint into a spectator sport. See methodology.

If you do run a public board, publish the protocol first and the trophies second.

Design patterns that transfer across domains

Policy stress pattern

Adversarial physics tiers (mass/friction and friends), paired seeds, Wilson CIs, shown severe-tier failures. Haga applies this on robosuite Lift, Stack, PickPlaceCan, and Door. Public shape: lift stress lab.

World-model physics pattern

Real negative controls, held-out generative cohorts, trajectory-level checks, documented failure modes, discovery separated from confirmation. Haga’s Physics-IQ-oriented cohort: real quiet 0%; CogVideoX held-out 1.000 via static_hover (n=9, Wilson [0.701, 1.000]). Lab.

These are analogues: different artifacts, same refusal to self-grade quietly.

A minimal benchmark spec template

Copy/paste and fill:

Benchmark name:
Protocol version:
Artifacts under test: (policy | world model | stack)
Sim substrate:
Hardware substrate: (or "not in scope for v1")
Factors swept:
Primary metrics:
Failure taxonomy:
Seeds & splits:
Negative controls:
Reporting rules: (intervals, no pooling discovery+held-out)
Non-claims:
Change control: (who can edit protocol; how versions bump)

If any field is blank, you do not have v1 yet.

When hardware is scarce, narrow claims — don’t inflate adjectives

Allowed under sim-only constraints:

• “Under protocol v1, severe friction tiers reduce success from A→B with shown failure mode F.”
• “Generative cohort G fails VIDEO_CHECKS via mode M; real quiet controls remain quiet.”

Not allowed:

• “Production ready.”
• “Human-level household competence.”
• “Solved sim-to-real.”

The industry already knows transfer is hard — figures like 89.4% sim versus ~12% real household success exist for a reason. Benchmarks should illuminate that gap, not paper over it.

Role of an independent evaluator

Independence buys you:

• thresholds the authors cannot quietly loosen,
• failure galleries that survive marketing edits,
• and a shareable methodology pointer for partners.

That is Haga’s product lane: sim-first, private, dual-pillar verification for policies and generative worlds. Not a training loop. Not a simulator vendor. Not Cosmos/NIM. Not “trust us, watch the video.”

Publishing without overclaiming

When you release results:

1. Lead with protocol version and non-claims.
2. Show failures next to successes.
3. Keep discovery and held-out math separate.
4. Link raw enough artifacts for audit.
5. Update the date when the protocol changes — not when the narrative does.

Worked example: pairing a policy score with a world-model score

Suppose your stack trains a pick-and-place policy partly on generative augmentations. A defensible sim-to-real program (not a single number) might report:

1. World-model packet: protocol v1 VIDEO_CHECKS on held-out generative samples; real quiet controls at 0%; documented mode static_hover with interval.
2. Policy packet: protocol pickplace-stress-v3 across mild/moderate/severe mass×friction; grasp-integrity separated from place-stability; severe-tier failure gallery.
3. Hardware packet (when available): same taxonomy on N real SKUs; no averaging into the sim score.

Readers can then ask the only question that matters: which layer is the bottleneck? Benchmarks that force that question accelerate progress. Benchmarks that hide it accelerate fundraising theater.

Anti-patterns to ban in v1

• Silent threshold edits after peeking at the test set.
• “Pass” badges without severe tiers.
• Pooling discovery samples into the confirmatory mean.
• Rankings that omit seeds and non-claims.
• Hardware adjectives attached to sim-only rows.

Write them into the protocol README as forbidden moves. Culture beats tooling here.

Next step

Designing a benchmark or need a private run against a fixed protocol? Start with Haga’s public methodology, inspect the lab, or request an eval.

Pass rates are easy. Reproducible trust is the actual product.