Generative video that looks right but breaks physics

Every major text-to-video and image-to-video release arrives with a demo reel that looks stunning and a set of physics failures that look hilarious. The joke clips — floating coffee cups, chairs melting into tables, hands that fuse with objects — are usually presented as quirks. They are not quirks. They are the most informative data the model ships.

A catalog of failure modes, collected under a fixed protocol, tells you more about a generative world model than any leaderboard of highlights.

The failure classes that actually appear

Across Sora, Veo, CogVideoX, and similar systems, physics failures cluster into a small set of reproducible classes:

1. Static hover (the freeze)

The model "solves" dynamics by freezing objects in place — a cup hangs mid-air because the model never decides to move it. This is the single most common failure in image-to-video generation. It is exactly the failure class that dominated our Physics-IQ held-out cohort: CogVideoX I2V scored 100% flags via static_hover (n=9, Wilson CI [0.701, 1.000]) while real quiet video scored 0%.

2. Teleportation

Objects jump between poses across frames with no visible trajectory. Common when occlusion breaks tracking: an object hidden behind a hand re-appears in a different position. A teleporting object is a world with discontinuous state — useless for planning, training, or simulation.

3. Interpenetration

Rigid bodies pass through each other. The classic is a hand closing through a cup. Interpenetration is a constraint violation, and unlike hover or teleport it survives longer clips: the model keeps rendering a physically impossible contact for many frames.

4. Anti-gravity and impulse artifacts

Objects accelerate without contact, or change velocity discontinuously — an object "launched" by a hand that never touched it. Detectors look for velocity discontinuities and acceleration without contact sources.

5. Context amnesia

Objects change identity, color, or count across the clip. Fewer people in frame than at the start; a red mug becoming blue. This is less a physics failure and more an object-permanence failure, but it breaks the same downstream use cases.

Why highlights hide all of this

Demo reels select clips that render well. Selection pressure means the public never sees the failure distribution. A model that fails 40% of the time can ship a reel with zero visible failures — the reel is a sample, not a rate.

Independent evaluation inverts the incentive: score every output in a cohort under a frozen protocol, then report the distribution. The world-model evaluation metrics post explains why distributional metrics like FVD miss these failures entirely.

How to turn failure modes into evidence

1. Freeze the protocol first. Seeds, prompts, detectors, and thresholds are fixed before the cohort is generated.
2. Score a cohort, not a reel. Twenty outputs per condition minimum; report the flag rate per failure class.
3. Calibrate the detectors. Run them on real quiet video and hand-built violations before trusting any cohort number. Haga's physics checker publishes recall and false-positive rates as a precondition for scoring.
4. Publish the intervals. n is small in practice; Wilson intervals keep people from narrating noise as progress.
5. Separate the claim. "Realistic render" and "physically consistent" are different claims with different evidence. Say which one you are making.

What a credible report looks like

A credible generative-video evaluation names the model version, the exact prompt set, the detector definitions, the calibration numbers, and the cohort size — then shows the failure distribution per class. That is the format used in the held-out Physics-IQ writeup and the technical article.

When a vendor shows you a demo reel instead of a flag-rate table, you are looking at marketing. Ask for the failure distribution; the answer tells you whether the model is a renderer or a world model.

Building the cohort without leaking your own bias

The failure catalog is only as good as the cohort that produced it, and cohort construction is where evaluations quietly go wrong. Three rules keep the cohort honest:

1. The cohort is frozen before the first output

Write the prompt set, the seed list, and the detector versions into version control before generating anything. The moment you adjust the prompts after seeing outputs, you are fitting the cohort to the model — and the "held-out" label is a fiction. The Physics-IQ held-out cohort documents this discipline in practice: the protocol and thresholds were fixed before the model outputs were scored, which is precisely why the numbers can be cited.

2. Conditions come from the claim, not the model

Every condition should map to a claim the vendor is likely to make: "handles occlusion", "realistic object permanence", "consistent contact". You pick the claims first from the marketing materials, then build conditions that stress exactly those claims. If you pick conditions after seeing what the model does well, you have inverted the process — you are searching for evidence of strength instead of testing for weakness.

3. Controls run alongside

Every cohort run needs a control condition: real, quiet, physics-known video scored through the same detectors with the same thresholds. The control tells you the detector's baseline false-positive rate in situ. Without a control, a 15% flag rate on the model is uninterpretable — it might be model failures, or it might be the detector firing on plausible-looking video. On the held-out cohort, the real-video control scored 0% flags while the generative condition scored 100% via static_hover — the contrast between control and treatment is the finding, not the model's number alone.

Failure classes are a product spec

Here is the shift that matters for teams building on generative video: the failure catalog is not a bug report, it is a capability spec. A world model with zero static_hover but 20% interpenetration is a model that understands dynamics but not constraints. A model with the inverse profile understands constraints but not dynamics. Those are different products with different downstream uses — one might serve as a planning prior, the other as a renderer. A single pass rate cannot tell you which model you are holding, but the per-class distribution can. This is the same reasoning that pushes manipulation evaluation toward per-layer metrics: aggregation hides which subsystem needs the fix.