World model evaluation: from plausible clips to reproducible claims

World models are having a moment. Text-to-video and image-to-video systems produce clips that survive a casual watch. Robot-learning teams experiment with generative worlds as training fuel. Executives ask whether the model “understands physics.”

Most evaluation stacks still answer a different question: whether the pixels look like other pixels.

That mismatch is why world-model evaluation has to grow up — from vibe checks to reproducible claims with explicit boundaries.

The limits of looking good

Metrics in the FID / FVD family (and their cousins) are useful for distributional similarity. They are weak instruments for physical understanding. A model can match motion statistics and still:

• freeze an object mid-air (static_hover-style failures),
• teleport contact points,
• invent anti-gravity hops,
• interpenetrate rigid bodies,
• or break ballistic continuity under fixed seeds.

Physics-IQ made this scientific debate concrete: visual realism does not imply physical understanding. Subsequent “verified” refinements of the benchmark emphasize cleaner prompts, fewer artifact-driven metric activations, and sample-aware scoring — because measurement error becomes policy when leaders ship on the score.

Haga’s job is adjacent but product-shaped: private verification of generative world-model outputs with detectors and trajectory checks teams can use as an acceptance gate.

What “world model evaluation” should mean in 2026

A serious evaluation program for generative or learned worlds should measure at least four things:

1. Physical consistency under fixed initial conditions

Given the same conditioning inputs, does the continuation obey coherent object permanence, contact, and motion constraints? If your export is video, evaluate the video interface — not a hidden simulator state the customer will never receive.

2. Failure modes, not only averages

A mean score without a taxonomy teaches nothing operational. Teams need names: freeze/hover, teleport, impulse spikes, interpenetration, unsupported hovering, impossible collisions.

3. Negative controls

Real quiet or physically simple real footage should not trip your detector. If it does, you built a noise amplifier.

4. Claim boundaries

Separate:

• “useful as a training prior under protocol X,”
• “consistent enough to cite in a paper under protocol Y,”
• and “ready to underwrite hardware decisions.”

Those are different products of evaluation. Collapsing them is how trust dies.

A practical stack: detectors, then trajectories

Haga’s world-model pillar follows a deliberate order:

1. Calibrated physics-violation detectors for classes like teleport, anti-gravity, impulse, and interpenetration.
2. CoTracker3 → VIDEO_CHECKS on position sequences for generative cohorts where the interface is video.

On calibrated checker v0, physics-violation detection reported 1.000 recall with 0 false positives on the calibrated suite. That is a detector claim, not a universal certificate for every future generator.

On Physics-IQ-oriented work, real quiet controls flagged at 0%, while a CogVideoX I2V held-out cohort (n=9, protocol v1) flagged at 1.000 via static_hover (Wilson 95% CI [0.701, 1.000]). Discovery (n=6, post-hoc) stayed separate from held-out confirmation. Read the lab and methodology.

This is what “state of the art” should mean for diligence: boring protocols, shown failures, intervals, and non-claims — not a bigger adjective in a launch blog.

Why paired seeds and protocol freezes matter

Generative systems are stochastic. If you resample until you find a pretty clip, you are not evaluating a model. You are evaluating your patience.

Evaluation discipline looks like:

• paired seeds across conditions,
• frozen preprocessing,
• documented prompt or conditioning templates,
• held-out splits that do not leak discovery edits,
• and a refusal to silently change thresholds after peeking.

If your team cannot re-run last month’s protocol bit-for-bit, you do not have a benchmark. You have a scrapbook.

Where world-model eval sits relative to policy eval

World-model physics QA and policy stress tests answer different questions:

Artifact	Core question	Example Haga posture
Generated / learned world	Is the world physically consistent enough to trust as a prior?	Physics-IQ-style VIDEO_CHECKS, detector suite
Policy / controller	Does the policy survive adversarial physics tiers?	Mass/friction stress on robosuite Lift, Stack, PickPlaceCan, Door

Training a strong policy inside an inconsistent world can produce impressive internal scores and miserable transfer. Evaluating only the policy hides the world’s contribution to the failure.

Common mistakes we see in world-model decks

1. One cinematic clip as proof. Evidence, not proof.
2. Leaderboard cosplay without protocol. Rankings without seeds and controls are entertainment.
3. Mixing FVD improvements with physics claims. Different axes.
4. Pooling discovery and confirmatory sets. Inflates confidence.
5. Skipping real negative controls. Detectors that hate reality are useless.
6. Equating video QA with robot readiness. Hardware still has to earn its own layer.

What to put in an eval packet reviewers will respect

If you are preparing diligence materials for partners or investors, include:

• protocol version and date,
• model/checkpoint identity and conditioning mode (I2V, T2V, etc.),
• sample sizes with intervals,
• negative control results,
• failure-mode gallery with trajectories or overlays,
• explicit non-claims,
• and a pointer to independent methodology when available.

Haga can produce that packet privately. We are not a public leaderboard, not Cosmos/NIM, and not a training platform. We are the verification layer.

Getting started

• Skim the public methodology.
• Inspect Physics-IQ lab evidence.
• For a private generative cohort, request an eval.

Plausible clips will keep getting cheaper. Reproducible physics claims will not — unless you build evaluation like you mean it.