World model evaluation metrics: why FVD isn't physics

When teams report a world model's quality with a single number, that number is usually FVD (Fréchet Video Distance) or a cousin like FID. The number is defensible for one question and one question only: does the generated video look distributionally similar to the reference set?

That is a real question. It is not the question release engineers are asking.

A model can match motion statistics and still violate physics in ways that break downstream use: objects frozen mid-air, contact points teleporting, rigid bodies interpenetrating, trajectories that ignore gravity. FVD is blind to all of it. The Physics-IQ benchmark made this concrete: visual realism and physical consistency are separable properties, and scoring a model on one says nothing about the other.

Why distributional metrics saturate and mislead

FVD-style metrics compare sets of video frames or clips. The score is sensitive to texture, lighting, camera motion, and content composition — all the things that make video look "real." Physical violations are sparse events: a 3-second clip with one interpenetration frame still scores close to a clean clip.

Worse, distributional metrics reward average behavior. A model that occasionally freezes objects but otherwise renders plausible motion will beat a model that is always correct but slightly less crisp. For world models used as training fuel or simulation substitutes, that ranking is inverted from what you actually care about.

This is not a criticism of the metric's math. It is a scope warning. FVD answers "do these pixels resemble those pixels?" — and nothing else.

What to measure instead

1. Trajectory-level physical checks

Instead of comparing sets, check each output against physical invariants:

• Contact integrity: do objects remain rigid through contact events?
• Gravity/ballistics: does ballistic motion follow expected trajectories?
• No-interpenetration: do rigid bodies pass through each other?
• Motion continuity: are there teleports, jumps, or frozen frames under fixed seeds?

The key property is deterministic, seedable, and replayable — the same inputs must produce the same verdict.

2. Calibrated violation detectors

Detectors need a calibration story. Run them on known-clean real video and measure false positives; run them on hand-built violations and measure recall. A detector that flags 30% of real quiet video is noise, not evidence. Haga's physics checker publishes exactly this: recall and false-positive rates under noise control before any cohort is scored (see the physics checker lab).

3. Held-out cohorts with claim boundaries

The evaluation protocol should freeze before the cohort is collected, and the held-out set should never touch training, prompting, or tuning. Without this discipline, the "benchmark" measures prompt engineering, not world-model quality. The Physics-IQ held-out cohort is a worked example: real quiet controls scored 0% flags while a generative model scored 100% via a single detector class.

4. Failure taxonomies over pass/fail

"Did it pass?" is the wrong headline. "Which violation classes fired, at what rates, with what intervals?" is the right one. static_hover, teleport, interpenetration, and impulse are different bugs with different fixes. A single composite score hides them all.

The metrics that survive a skeptic

The evaluation stack above has one property the FVD stack lacks: auditability. Anyone can re-run the checks, re-score the cohort, and reproduce the numbers. When a skeptic asks "what exactly did you measure?", a metrics suite that names detectors, seeds, cohorts, and thresholds answers in one sentence.

Distributional metrics have a place — as a cheap screening signal during training, not as a release gate. For physical AI, the release gate belongs to physics checks, calibrated detectors, and held-out cohorts. We wrote the fuller argument in world model evaluation: clips vs reproducible claims and publish live protocol numbers on the Lab.

A worked protocol you can copy

A concrete, defensible protocol has six steps:

1. Freeze the cohort. Twenty prompt-condition pairs per condition class, selected and locked before any model output is generated. The cohort definition, the seeds, and the scoring script go into version control on day one.
2. Calibrate the detectors. Run each detector on a known-clean real-video set and a hand-built violation set. Report recall and false-positive rate before scoring anything. A detector with high false positives will flag real video and inflate model failure rates; a detector with low recall will clear violators. Both failure modes are silent without calibration.
3. Run the model once. One forward pass per cohort entry with pinned seeds. No resampling, no cherry-picking, no "rerun with a better prompt."
4. Score mechanically. Detector verdicts per frame and per clip, aggregated per failure class. Human review only to adjudicate boundary cases, with the adjudication logged.
5. Report the distribution. Flag rate per class with Wilson intervals, n per condition, and the full confusion against calibration sets. Do not report a single pass rate.
6. Publish everything. Seeds, prompts, detector versions, thresholds, and raw verdicts. Reproducibility is the property that separates measurement from narration.

Where intervals matter most

Small cohorts are unavoidable in world-model evaluation — each output is expensive. The discipline that keeps small n honest is the Wilson interval. With n=9 and 9 flags, the rate is 100% with an interval of roughly [70%, 100%]. With n=20 and 18 flags, 90% carries an interval of [70%, 97%]. Reporting the interval forces the reader to see that "90%" and "100%" are not the claims they appear to be — and prevents teams from narrating 1-in-20 noise as a 5-point improvement. Every rate on the Physics-IQ held-out cohort carries its interval for exactly this reason: small-n honesty is a reporting rule, not a statistics course.

The scorecard

When reviewing a world-model evaluation, ask for these five artifacts in order:

• the frozen cohort definition (with git history),
• detector calibration numbers (recall + false positives on known sets),
• per-class failure rates with intervals and n,
• the raw verdicts (not just aggregates),
• the exact commands to re-run everything.

An evaluation that produces all five is rare — and that rarity is the point. The field's next leaderboard wars will be won by whoever treats world-model metrics as an audit trail rather than a scoreboard.