Physics-IQ

Held-out confirmation: freeze/hover still fires after protocol freeze.

Discovery (seeds 0–1, n=6) adapted `static_hover` after inspecting those clips — post-hoc, not confirmatory. Held-out protocol v1 froze thresholds first, then scored seeds 2–4 × left/center/right (n=9): flag rate 100% [0.701, 1.000], all `static_hover`. Real Physics-IQ negative control stays quiet. Held-out protocol v1 (cogvideox-heldout-v1): CogVideoX I2V on Physics-IQ ball-and-block-fall; n=9 (center/left/right, seeds 2, 3, 4); frozen VIDEO_CHECKS; separate from post-hoc discovery n=6. Not Cosmos/NIM. Not a broad generative-quality verdict.

What we tested

Does frozen VIDEO_CHECKS stay quiet on real clips and confirm freeze/hover on a pre-registered CogVideoX held-out matrix?

real negative control

100%

held-out flag rate (n=9 [0.701, 1.000])

static_hover

failure mode (all 9 clips)

Charts

Flag rate by cohort (discovery vs held-out)

flag rate

Methodology

Freeze detector + matrix. Lock VIDEO_CHECKS / static_hover thresholds and the held-out seed×scenario matrix before generating confirmatory clips.
Track + score cohorts. Run CoTracker3 on real Physics-IQ, discovery, and held-out CogVideoX folders with the same frozen profile.
Publish separately. Report discovery and held-out flag rates with Wilson CIs; document failure mode (static_hover) without pooling cohorts.

Seeds: CogVideoX discovery: seeds 0, 1, center/left/right (n=6); CogVideoX held-out: seeds 2, 3, 4, center/left/right (n=9)

real clips expected quiet for this scene
held-out matrix scored only with frozen VIDEO_CHECKS
generative cohorts document static_hover when flagged

Public methodology

Notes

CogVideoX = THUDM/CogVideoX-5b-I2V, not Cosmos. held-out n=9 (center/left/right, seeds 2, 3, 4); `static_hover` documents one generative failure mode — not a broad generative-quality verdict. Discovery cohort remains labeled separately (CogVideoX = THUDM/CogVideoX-5b-I2V, not Cosmos. discovery n=6 (center/left/right, seeds 0, 1); `static_hover` documents one generative failure mode — not a broad generative-quality verdict.)

One freeze/hover failure mode on CogVideoX ball-and-block-fall — not Cosmos, not a broad generative-quality verdict. Discovery remains post-hoc; cite held-out for confirmatory claims.