Mobile robot navigation evaluation: success rate is not enough

A mobile robot that reaches the goal in a tidy map can still be one sensor glitch away from a bad day. Navigation teams know this in their bones. Evaluation practice often forgets it anyway — collapsing everything into a single success rate that flatters the lab layout.

If you want claims that survive contact with hallways, warehouses, and humans, you need failure-aware navigation evaluation.

The success-rate trap

Goal-reached percentage answers one question: how often did the trial end in the target region under the conditions you ran?

It does not answer:

• How badly did localization drift before recovery?
• What happens under lidar dropouts or camera occlusion?
• Does obstacle behavior remain conservative when dynamics change?
• Is performance an artifact of one memorized map?
• Are near-misses counted, or only hard collisions?

A stack can post a high success rate while skating on thin localization and brittle costmaps. That is not robustness. That is a quiet debt.

Break navigation into subsystems you can blame

Localization stability

Measure pose error growth under odometry noise, wheel slip proxies, and intermittent sensing. Track time-to-divergence and time-to-recovery separately from goal success.

Mapping / prior quality

If you rely on a prior map, ablate staleness: moved furniture, altered shelves, temporary obstacles. If you rely on SLAM, ablate loop-closure stress — not only the happy path.

Planning

Separate global plan feasibility from local control tracking. A perfect global path is worthless if the local controller chatters into obstacles.

Obstacle behavior

Log near-misses, emergency stops, and clearance distributions — not only binary collisions. Conservative stacks and aggressive stacks can share a success rate and differ entirely in operational risk.

Randomize worlds; freeze protocols

A single floorplan is a story. A distribution of worlds is a test.

Evaluation upgrades that pay off quickly:

1. Procedural or curated world variations with declared difficulty tiers.
2. Sensor dropout schedules applied identically across policies.
3. Paired seeds so A/B comparisons are meaningful.
4. Protocol IDs that bump when maps, sensors, or thresholds change.
5. Held-out layouts that never appear in training montages or weekly demos.

The goal is not infinite randomness. The goal is enough structured adversity that “it worked in the lab” stops being a free claim.

What to put in a navigation eval report

Reviewers should be able to find:

Section	Contents
Claim	Exact operational statement (and non-claims)
Environment distribution	Maps/worlds, tiers, held-out splits
Sensing assumptions	Modalities, dropout model, calibration notes
Metrics	Success, collision, near-miss, localization error, time
Failure taxonomy	Localization loss, planner infeasibility, control tracking, timeout
Statistics	n + intervals; avoid orphan percentages
Evidence	Trajectories, videos, or logs for representative failures

If the report only has a bar chart labeled “Success,” send it back.

Sim-first still needs honesty about the gap

Simulation is the right place to multiply dropout schedules and world variants. It is the wrong place to launder deployment adjectives.

Keep the claim ladder explicit:

1. Sim protocol evidence under stress.
2. Limited hardware routes that reuse the same taxonomy.
3. Pilot claims only for the sites and sensor kits you actually ran.

Physical AI’s broader trust problem — high sim scores, weaker real-world outcomes — shows up in navigation too. Treat Stanford AI Index-style gaps as a cultural warning, even when your task is hallways rather than household manipulation.

Where world models sneak into navigation

Learned world models and generative scene augmentation increasingly appear in navigation research and synthetic data pipelines. If generated video or neural scenes influence training, run physics-consistency checks on that prior. Visually calm footage can still hide freezes and impossible dynamics. Haga’s generative verification work — including Physics-IQ-oriented cohorts with quiet real controls and documented generative failure modes — exists for that layer. See Physics-IQ lab and methodology.

Navigation policies deserve teachers that do not invent physics.

Independence beats tribal thresholds

Internal dashboards drift. Someone loosens a collision buffer before a customer visit; someone else “temporarily” disables a dropout schedule. Independent evaluation freezes the protocol and returns shown failures — the same product logic Haga applies to manipulation stress tests and world-model QA.

We are not a mapping vendor, not a public navigation leaderboard, and not a replacement for your stack. We are a verification layer you can point partners at when success-rate slides are no longer enough.

Example stress schedule (adapt, don’t cargo-cult)

A lightweight v1 schedule for indoor mobile bases:

Tier	Sensing	World	What you hope to learn
A	Full sensors	Static map, light clutter	Baseline competence
B	20% lidar dropouts	Moved obstacles	Localization + costmap resilience
C	Camera occlusions	Narrow clearances	Conservative local control
D	Wheel-slip proxy	Held-out floorplan	Whether “success” was memorization

Run paired seeds across policies. Report clearance percentiles and localization error, not only goal-reached. Promote only the claims your worst tier still supports.

Connecting navigation eval to physical-AI diligence

Navigation is often sold as “solved infrastructure” while manipulation gets the research romance. Operationally, brittle navigation destroys pilots just as fast as brittle grasps. Apply the same cultural rules Haga pushes elsewhere: protocol versions, shown failures, intervals, and refusal to let a clean lab route impersonate site readiness.

Independent evaluation is how you keep those rules when internal incentives favor the pretty route replay.

Next step

If you need a private, protocol-bound stress packet for a navigation or embodied stack component, request an eval. For how we structure evidence publicly, start with methodology and the lab.