Physical AI verification: the security-audit model

The software industry solved a problem that physical AI is about to rediscover the hard way: self-certification does not work. In finance, security, and safety-critical software, the claim "we tested it" is not accepted without an independent party confirming it. The audit is a separate institution with its own incentives, its own protocols, and its own liability.

Robot and world-model teams are still in the "we tested it" era. The claims are getting bigger — deployable autonomy, physical-AI reasoning, general manipulation — while the evidence standard stays at the demo level. The fix is not better metrics. It is a different institutional structure: the security-audit model applied to physical AI.

Why self-reported benchmarks fail

The incentive problem is structural. The team that trained the policy:

• chose the evaluation environments,
• tuned hyperparameters against them,
• selected which runs to report,
• and defines the failure taxonomy.

Every step is rational. Together they mean the evaluation and the artifact share a single optimizing agent. Whatever the benchmark measures, the team has been optimizing it. The benchmark gaming post catalogues the failure modes this produces — overfitting, selective reporting, protocol drift — and they are all downstream of one root cause: the checker and the checked are the same entity.

The audit structure, translated

A security audit has five properties that make it trustworthy. Each translates directly to physical AI verification:

1. Separated checker

The auditor did not build the artifact. Independence means the checker does not benefit from the claim being true. For physical AI: the evaluation protocol, detectors, and scoring are owned by a party with no stake in the policy's success.

2. Frozen protocols

Audits run against published, versioned procedures. If the procedure changes between audit and claim, the claim is void. Physical AI's version is pinned seeds, fixed perturbation tiers, versioned detectors, and a scoring script that anyone can re-run. Our methodology is published on exactly this principle.

3. Adversarial testing

Auditors do not test the happy path; they probe the edges. The analog for physical AI is adversarial stress testing: mass and friction tiers past nominal, sensor noise, degenerate conditions — with degradation curves reported, not just pass rates.

4. Evidence over assertion

Audit reports show the work: logs, failure cases, intervals, reproductions. Physical AI's version is shown failure modes with thresholds, Wilson intervals, and cohort-level distributions — the format used in the Physics-IQ held-out cohort.

5. Claim boundaries

Audits certify what they audited — no more. A physical-AI verification report should state explicitly what was not tested: hardware-in-the-loop behavior, deployment site conditions, unseen object classes. Claim boundaries are not weakness; they are what makes the claim credible.

What changes for buyers and investors

The security-audit model changes the diligence question from "show me your numbers" to "show me who checked them."

• For investors: the same diligence instinct that demands audited financials should demand independently verified capability claims. A sim-to-real benchmark self-reported by the team is a pitch deck; a verification report from a separated checker is evidence.
• For operators: release gates should reference protocol versions and independent scores, not demo videos. Hardware pilot decisions hinge on degradation curves, which only adversarial evaluation produces.
• For labs: independent verification is not an accusation; it is the mechanism that makes your claims citable. The strongest teams will be the ones that invite the check.

The institutional shape

The audit model does not require a new bureaucracy. It requires a new default: evaluation as a separated function, with published protocols and adversarial methods. Haga exists to be that separated function for physical AI — independent verification of policies and world models, with the methodology, thresholds, and failure cases public (see the Lab for live evidence).

The software industry took decades to learn that "we tested it" is not a diligence standard. Physical AI can borrow the lesson instead of reliving it.

What a verification report must contain

Independence is the precondition, but the report itself has a shape. A verification report that a buyer, investor, or operator can act on contains seven sections, each auditable:

1. Scope and claim boundaries. What was verified, what was not: the exact policy/world-model version, the simulator and version, the task set, and an explicit list of untested claims (hardware-in-the-loop behavior, deployment-site conditions, unseen object classes). Boundaries are not weakness — they are what make the report citeable.
2. The frozen protocol. Seeds, perturbation tiers, detector versions, scoring scripts, and thresholds — versioned and public before the cohort ran. The methodology is published on exactly this principle.
3. Calibration evidence. For any detector: recall and false-positive rates on known-clean and hand-built-violation sets. Numbers without calibration are vibes.
4. Cohort results with intervals. Per-class failure rates, n per condition, Wilson intervals. The format used in the Physics-IQ held-out cohort.
5. Degradation curves. Performance under adversarial stress testing — nominal through extreme severity, with failure taxonomies per tier.
6. Raw artifacts. Episode logs, verdicts, seeds — enough to re-run the scoring, not just the summary.
7. Reproduction instructions. The exact commands and environment to reproduce the report from scratch.

A report missing any of these is a summary, not verification. The distinction matters because the market will start demanding the second kind — and the teams that already produce it have an asymmetric advantage.

Who verifies the verifier

The honest objection to the audit model is: who verifies the verifier? Three answers, in order of strength:

1. Reproducibility is the check. A verifier whose protocols, artifacts, and commands are public can be re-run by anyone — the buyer, the lab, a competitor's engineer. Public reproducibility converts the verifier's reputation into a shared asset: any error is discoverable, so errors get corrected. This is the same pressure that keeps open-source security audits honest.
2. No conflict of interest. The verifier must not be paid by the claim's success — no success fees, no equity in the lab, no consulting relationship that bends the thresholds. Separated incentives are the entire point, as the benchmark gaming post argues about leaderboards generally.
3. Error audits. A credible verifier publishes its own calibration and, where applicable, its mistakes: protocol revisions are versioned with changelogs, threshold changes are public, and cohort adjustments are diffed in version control.

The model does not require the verifier to be infallible. It requires the verifier's work to be publicly checkable — which is what makes the independence structural rather than rhetorical. Any institution that wants to be the trusted checker for physical AI should expect to be checked itself, at full fidelity, forever.