Verification
Physics-violation detection: current benchmark results
100% detection of injected physics violations at zero false flags on clean runs — with the full methodology, thresholds, and confidence intervals published.
We are publishing our current benchmark results as raw numbers, not a case study. Every figure below is generated by the haga-core pipeline and traceable to versioned JSON artifacts (pillar1-*.json, pillar2-*.json, git SHA c848aab). If you want to verify any claim, ask for repo access.
What Haga checks
Two pillars:
- Policy stress gates — run a manipulation policy (Franka Panda) through nominal, mild, and moderate physics perturbation tiers; gate deployment on success-rate retention and grasp-failure deltas.
- Physics-consistency checking — score trajectories and AI-generated video against physical law: object permanence, ballistic motion, contact consistency, static hover, impulse limits.
Pillar 2: physics checker — synthetic violations
Injected violation classes, n=100 each:
| Violation class | Flagged | Rate | 95% CI |
|---|---|---|---|
| Teleportation | 100/100 | 1.00 | [0.963, 1.0] |
| Hover (unsupported objects) | 99/99 | 1.00 | [0.963, 1.0] |
| Impulse anomalies | 100/100 | 1.00 | [0.963, 1.0] |
| Penetration | 100/100 | 1.00 | [0.963, 1.0] |
Negative controls (must NOT fire):
| Cohort | Flagged | Rate |
|---|---|---|
| Clean trajectories | 0/100 | 0.00 |
| Clean + sensor noise | 0/100 | 0.00 |
Recall 1.00, precision 1.00 on this suite. The negative controls matter more than the recall: a checker that cries wolf is worse than useless as a deployment gate.
Pillar 2: generative video — held-out cohort
Model: CogVideoX-5b-I2V, tracker CoTracker3, failure mode: static_hover.
| Cohort | n | Flagged | Rate | 95% CI |
|---|---|---|---|---|
| Real reference footage | 1 | 0 | 0.0 | [0.0, 0.79] |
| CogVideoX generations | 9 | 9 | 1.00 | [0.66, 1.0] |
All 9 held-out generations across 3 unseen seeds were flagged; the real clip passed clean. Small n — we publish it anyway, because a confidence interval that admits uncertainty is worth more than a round number that hides it.
Pillar 1: policy stress gates
Four manipulation tasks, 50 episodes per condition, Franka Panda:
| Task | Success (nominal) | Success (mild perturbation) | Gate verdict |
|---|---|---|---|
| Lift | 1.00 | 1.00 | PASS (score 1.00) |
| PickPlaceCan | 1.00 | 1.00 | PASS (score 1.00) |
| Stack | 0.96 | 0.82 | PASS (score 0.854) |
| Door | 0.60 | 0.64 | PASS (score 1.067) |
The Door task shows what these gates are for: nominal success of 0.60 looks acceptable until severe-perturbation runs expose a 0.48 failure rate under load. A gate that only measures nominal success would have waved it through.
Honest scope statement
- Results are simulation-based (Panda manipulation tasks, synthetic violation injection, one generative video model).
- The checker's thresholds, check definitions, and scoring criteria are in the artifacts; we change them via versioned commits, never silently.
- We have no customer pilot results to report yet. When pilots exist, they will be published here with the same artifact discipline — or not published at all.
- Physical-robot validation is the next milestone, not a completed one.
If your team ships policies or world models into the real world, we'd like to run this suite against your stack.
Quick links
- One-pager (shareable/printable): https://haga.mushoodhanif.com/one-pager/
- Book a 15-min call: https://haga.mushoodhanif.com/book-call/
- Evidence page (this page): https://haga.mushoodhanif.com/blog/physics-violation-detection-benchmark-results
Contact: haga@mushoodhanif.com.
Want an independent check on a policy or world-model artifact? Haga runs fixed protocols with shown failures — not self-graded demos.
Related reading
- Obstacle avoidance metrics beyond collision counts
Zero collisions can still mean a robot that freezes, brushes, or deadlocks. Here is the avoidance metric stack that distinguishes safe navigation from lucky navigation.
- Physical AI verification: the security-audit model
The software industry solved self-certification with independent audits. Physical AI verification should borrow that structure: separated checkers, frozen protocols, and adversarial evidence.
- Manipulation metrics that survive a skeptic
A skeptic asks: did the grasp hold, did placement land, did recovery terminate? Here is the manipulation metric stack that answers without being gamed.