Physics-IQ held-out cohort: real 0%, CogVideoX 100% via static_hover

Generative video models are getting better at looking real. That is not the same as understanding physics. Physics-IQ exists because distributional video metrics can reward plausible motion while missing broken contact, frozen trajectories, and physically impossible continuations. Haga’s held-out cohort is a narrow, protocol-bound check on that gap — not a universal ranking of world models.

This post documents what we ran, what failed, and what we refuse to claim.

Why visual realism is the wrong acceptance gate

World-model builders and robot-learning teams increasingly train on, or condition on, generated video. If the generator invents freezes, teleports, or anti-gravity contacts, downstream policies can overfit to artifacts that never survive contact with the physical world.

Perceptual scores help you ask “does this look like a video?” They do not answer “did the tracked objects obey a coherent physical story under fixed initial conditions?” Physics-IQ’s design pushes evaluation toward the second question by comparing generations against controlled real-world phenomena rather than only unmatched sets of clips.

Haga’s posture is sim-first and claim-bounded: we treat physics-consistency verification as a first-class evaluation problem with a fixed protocol, paired inputs, and an interface that matches what pipelines actually export — position-bearing video, not a private simulator state dump.

Protocol snapshot (v1 held-out)

Axis	Setting
Protocol	Physics-IQ Verified-oriented verification, Haga protocol v1 held-out
Negative control	Real quiet videos
Positive condition	CogVideoX image-to-video held-out subset
Sample size (held-out)	n=9
Tracking	CoTracker3 cohesive position sequences
Checks	VIDEO_CHECKS on tracked trajectories
Documented failure mode	static_hover
Real quiet flag rate	0%
CogVideoX held-out flag rate	1.000 (Wilson 95% CI [0.701, 1.000])

The negative control matters as much as the positive condition. A detector that fires on real quiet footage is not useful for diligence. In this cohort, real quiet stayed quiet.

What static_hover actually means

static_hover is not a vibes label. Under the tracked frames we verified, the generative sequences exhibited freeze/hover behavior: camera motion, objects, and contacts did not evolve coherently across the continuation. The failure is discoverable from position trajectories — the same interface many world-model and VGM pipelines expose after generation.

That makes the result operationally useful: you can reject or quarantine a batch without needing privileged access to the model’s training loop or a proprietary renderer state.

It also keeps the claim honest. We did not show that every CogVideoX sample in the wild fails this way, that every world model shares the same mode, or that fixing static_hover alone yields a physically trustworthy generator.

Discovery vs held-out: do not pool them

Before the held-out confirmatory run, we saw static_hover in a smaller discovery set (n=6, post-hoc). Discovery is for finding failure modes. Held-out is for checking whether the mode still appears under a frozen protocol.

We treat those as separate chapters:

• Discovery: hypothesis formation.
• Held-out protocol v1 (n=9): confirmatory measurement under fixed rules.

Pooling discovery and held-out into one “overall flag rate” would inflate confidence and blur the scientific boundary. The number we stand behind for protocol v1 is the held-out flag rate of 1.000 with Wilson interval [0.701, 1.000] — wide because n is small, which is exactly why we refuse to oversell.

How tracking enters the story

The evaluation path is deliberately boring:

1. Obtain the video under the protocol’s initial conditions.
2. Extract cohesive position sequences with CoTracker3.
3. Run VIDEO_CHECKS for physics-consistency violations on those trajectories.

Earlier calibrated checker work (v0) on physics-violation detection reported 1.000 recall with 0 false positives on the calibrated suite — teleport, anti-gravity, impulse, and interpenetration style failures. That checker work is not the same claim as the CogVideoX held-out cohort; it is part of the same verification stack. See the public methodology and Physics-IQ lab results for the current evidence layout.

Interpreting the Wilson interval without self-deception

A point estimate of 1.000 on n=9 is dramatic and incomplete. The Wilson interval [0.701, 1.000] says: under this protocol, we have strong evidence of a high flag rate, and we still have uncertainty about how high.

That is the correct diligence posture. Independent evaluation should make uncertainty visible. Self-graded demos hide it.

Claim boundaries (read these before you cite us)

This protocol measures physical consistency of generated video under fixed initial conditions for one generator family, one dataset interface, one protocol version, and one documented failure mode.

It does not:

• Certify hardware readiness, safety cases, or pilot readiness.
• Rank “all world models” or replace Physics-IQ’s public leaderboard methodology.
• Prove generalization of the detector without recalibration.
• Equate simulation-only validation with real-world robot behavior.
• Claim Cosmos, NIM, or any other stack we did not run.

If you need a broader statement, you need a broader protocol — and more held-out evidence.

Why this matters for agents and policies

Agents trained on visually plausible but physically inconsistent worlds can learn shortcuts: hover as if contact resolved, freeze as if dynamics paused, ignore momentum because the generator did. Those shortcuts look fine in a highlight reel and collapse when mass, friction, or true contact appears.

Independent verification is how you catch that early. Fix generation, change acceptance gates, or narrow the claim — but do not ship a self-graded “physics-aware world model” headline on the strength of a few cinematic clips.

What teams should copy from this cohort

If you are building an internal world-model gate, steal the structure, not just the slogan:

1. Fixed protocol version with frozen preprocessing.
2. Real negative controls that must stay quiet.
3. Held-out confirmatory sets separated from discovery.
4. Trajectory-level checks aligned to the export interface you actually ship.
5. One documented failure mode at a time, with explicit non-claims.
6. Intervals, not vibes — especially at small n.

Haga’s product shape matches that structure: private, sim-first physics verification for generative world-model outputs and for policies trained under physics stress. We are not a public leaderboard, not a simulator platform, and not a training loop.

Next steps

• Inspect the public write-up on methodology.
• Review the lab evidence for Physics-IQ runs.
• If you have a generative cohort or world-model export to check, request an eval.

The goal is not to declare Physics-IQ “solved.” The goal is to make physics failures visible early enough that teams can change what they trust.