Sim-to-real trust gap: why physics trust is the real bottleneck

Teams usually describe the sim-to-real gap as a hardware or domain-randomization problem: make the simulator prettier, randomize more parameters, buy better sensors, and transfer will follow. Sometimes that helps. Often the deeper failure is quieter — nobody independent checked the physics assumptions inside the test.

That is a trust gap. It shows up in board decks as a single success-rate number and in the lab as a brittle policy that only works under the author’s favorite seeds.

The number that should make you uncomfortable

Stanford’s AI Index (2026) reports a brutal split: roughly 89.4% success on simulation manipulation benchmarks versus about 12% success on real household tasks. Those are not two measurements of the same claim. They are evidence that “works in sim” and “works where humans live” have been allowed to share a marketing vocabulary.

If your release process treats them as interchangeable, you do not have a transfer strategy. You have a narrative strategy.

Self-grading is the hidden failure mode

In many robotics orgs, the same team:

• chooses the simulator and assets,
• writes the evaluation script,
• tunes thresholds until the plot looks good,
• and then reports the pass.

That loop can be good engineering. It is bad diligence. Reviewers — investors, partners, safety stakeholders, or a future you six months later — cannot tell whether the policy is robust or whether the test was softened until it passed.

Independent evaluation changes the incentives:

• Reproducible thresholds instead of self-tuned passing conditions.
• Shown failures instead of only curated successful rollouts.
• Shared methodology so a third party can inspect evidence.
• Claim boundaries that separate sim evidence from hardware validation.

Haga exists for that missing layer: private, sim-first physics verification you can buy rather than rebuild.

Fidelity theater vs evaluation discipline

Buying a more cinematic simulator can improve training coverage and still leave evaluation weak. Cinematic lighting does not guarantee deterministic seeds, inspectable contact events, or a failure taxonomy anyone outside the team can audit.

Ask different questions:

Training-oriented question	Evaluation-oriented question
Does this look realistic enough to learn from?	Can we reproduce the exact stress condition?
Can we randomize textures and lighting?	Do mass/friction tiers change success with confidence intervals?
Can we generate endless rollouts?	Do we record how failures happen?
Can we beat last week’s internal score?	Would an outsider accept the protocol?

If you only optimize the left column, transfer will keep surprising you.

What independent physics evaluation actually changes

1. It forces protocol versioning

A result without a protocol version is a demo. A result with a frozen protocol, seeds, and world variations is evidence. Versioning also makes regressions visible: if severe-tier friction used to fail and suddenly “passes,” you want to know whether the policy improved or the test drifted.

2. It separates mild stress from severe stress

Haga’s policy pillar adversarially varies mass and friction on robosuite tasks such as Lift, Stack, PickPlaceCan, and Door. We report success with Wilson confidence intervals and show severe-tier failures — never a gate-only PASS. See lift stress lab results and the methodology.

Mild tiers tell you the policy can move. Severe tiers tell you whether contact assumptions were a costume.

3. It makes world models part of the same trust story

Policies are not the only artifact. If your stack trains inside generated or learned worlds, those worlds need physics QA too. Haga’s second pillar runs calibrated physics-violation detection and trajectory-level VIDEO_CHECKS on generative video — including Physics-IQ work where real quiet controls stayed at 0% flags while a CogVideoX held-out cohort flagged at 1.000 via static_hover. Details: Physics-IQ lab.

A trustworthy physical-AI stack needs both pillars. Fancy policy plots on a broken world are still broken.

Hardware scarcity is not an excuse for weak claims

Hardware is expensive, slow, and contested inside every lab. That is why sim-first evaluation is rational. It is not a license to pretend sim scores are deployment certificates.

A healthy claim ladder looks like this:

1. Sim protocol evidence under adversarial physics tiers.
2. World-model consistency evidence if generated/learned priors are in the loop.
3. Limited hardware-in-the-loop on the same task family.
4. Pilot claims only after the previous layers agree on failure modes — not just averages.

Skip steps, and you are exporting optimism.

Practical checklist for teams this quarter

Use this as an internal diligence gate before the next external demo:

1. Name the exact claim you want to make (policy robustness? world-model consistency? pilot readiness?).
2. Freeze a protocol version and refuse silent edits mid-campaign.
3. Report failure modes, not only success rates.
4. Include confidence intervals at small n; do not narrate luck as certainty.
5. Keep negative controls (especially for generative video).
6. Separate discovery from held-out confirmation.
7. Publish or share raw enough artifacts that a skeptical partner can follow the story.
8. If you cannot run hardware yet, narrow the claim instead of inflating the adjective (“production-ready”).

What Haga is — and is not

Haga is the verification layer: independent stress-testing of simulated worlds and robot policies before they ship into the real one.

Haga is not:

• a public leaderboard replacing academic benchmarks,
• a simulator platform,
• a training loop,
• or an endorsement of any single proprietary stack (including ones we did not evaluate).

If you want the public posture, read methodology. If you want a private run on your artifact, request an eval.

Bottom line

The sim-to-real gap will not be closed by prettier pixels alone. It closes when teams treat physics assumptions as audit objects: versioned, stressed, failed in public (or at least in a shareable packet), and bounded in language.

Trust is not a vibe you get from a demo reel. Trust is what remains after an independent protocol tries to break you — and shows the breaks.