Manipulation metrics that survive a skeptic

"Grasp success rate: 94%." It is the most common metric in manipulation papers, and it is also the one most likely to survive a demo and die in a deployment review.

The problem is not the number. It is the aggregation. "Grasp success" is doing too much work: it collapses motion errors, grasp integrity, placement precision, and recovery behavior into one pass/fail label. A policy that fumbles and recovers still "succeeds." A policy that grasps but drops during transport still "succeeds." A policy that places in the wrong bin still "succeeds."

Here is the metric stack that survives a skeptic — the one we use when verifying pick-and-place claims.

Split the task into evidence layers

1. Motion success

Did the arm reach pre-grasp, lift height, and target placement within tolerances? This layer passes while the object is already slipping. It is the weakest evidence on its own — but it is the layer where timing and trajectory violations show up.

2. Grasp integrity

Did the grasp remain stable through lift and transport? Track slip events, regrasps, and drops separately from end-effector pose error. A grasp that survives lift but drops mid-transport is a transport failure, not a grasp failure. The taxonomy matters because it says which subsystem to fix.

3. Placement precision

Did the object remain in the target region after release — including after a short settle window? "Released above the bin" is not "placed in the bin." Evaluate with a settle window and a placement tolerance, and report the distribution, not the mean.

4. Recovery behavior

If the first grasp fails, does recovery terminate cleanly or thrash until timeout? Timeouts are failures — label them as such. A policy that recovers in 2 of 10 failures and times out in 8 has a recovery story worth reporting, but only if the taxonomy makes it visible.

The numbers that carry weight

Within each layer, three reporting rules separate honest evaluations from demo-driven ones:

Report per-class rates, not a composite

A single "success rate" mixing all four layers hides the subsystem you should fix. Report per-layer: motion rate, grasp-integrity rate, placement rate, clean-recovery rate. A skeptical reader can then see exactly where the policy leaks.

Report degradation curves under perturbation

Nominal performance is the least interesting number. The policy's behavior under mass and friction perturbation — the tiered severity curves published on the Lab — is what predicts field performance. A policy that holds 94% at nominal and 30% at tier 3 is a different product from one that degrades to 70%. Adversarial stress testing is what produces those curves.

Report intervals with small n

n=20 with 18 successes is 90% with a Wilson interval of roughly [70%, 97%]. Teams that print the point estimate without the interval are narrating noise. The interval is not decoration; it is the difference between a claim and a measurement.

What about force and contact metrics?

Force and torque traces (peak end-effector force, impulse at contact) are the most informative continuous signals in manipulation — and the most ignored. A grasp that succeeds with 3x nominal force is a safety incident waiting for a field deployment. Add force-trace reporting whenever the simulator exposes it; robosuite and MuJoCo both do. This is why the Lab charts include peak-force series alongside success rates.

The skeptic's checklist

Before accepting any manipulation evaluation:

1. Is the task split into motion, grasp, placement, and recovery evidence?
2. Are failure modes named per class — slip, drop, misplace, timeout?
3. Are degradation curves under perturbation shown, not just nominal?
4. Are intervals reported with n?
5. Were the protocol, seeds, and thresholds frozen before the runs?

Manipulation is the hardest evaluation surface in robotics because the failures are contact physics — invisible in success-rate aggregates and invisible in demo reels. The metric stack above is what makes those failures countable. For the deeper argument on why demo-scale evidence is not diligence-scale evidence, see the embodied AI trust gap.

A worked example: scoring a pick-and-place policy

Put the stack to work on a concrete claim: "PickPlaceCan reaches 90% task success." Here is what the four-layer split reveals that the aggregate hides.

• Layer 1, motion: the policy reaches pre-grasp in 96% of trials. Two failures are trajectory timeouts — the planner stalls at the approach pose.
• Layer 2, grasp integrity: of the 96%, only 88% keep the can through lift and transport. Eight trials slip mid-transport — a transport failure that the aggregate classifies as "grasp failure" if it does, or "success" if the can is recovered in the bin.
• Layer 3, placement: 84% release within placement tolerance after the settle window. Four trials drop the can above the bin — outside the settle window and, in a lax protocol, silently counted as placements.
• Layer 4, recovery: on the 16% of trials that fail a layer, the recovery loop terminates cleanly in half and times out in half. The timeouts are the hidden cost — each one inflates task time by 3-4x.

The aggregate "90%" is a real number that describes none of these subsystems. The four-layer report describes all of them, and each layer points to a different fix: planner tuning for layer 1, grip force scheduling for layer 2, placement tolerancing for layer 3, and recovery policy for layer 4. This is exactly the breakdown behind the pick-and-place verification writeup and the tiered stress curves published on the Lab — where the same policy is then pushed through mass and friction perturbation tiers so the degradation curve, not the nominal rate, carries the claim.

Timeouts and recovery budgets

The most common way manipulation evaluations lie is through their time budget. A trial that ends in timeout is a failure that many protocols record as "aborted" and quietly exclude from the denominator. The rules that keep time honest:

• A timeout is a failure, full stop — it appears in the numerator of the failure rate and in the taxonomy under its own class.
• The time budget is part of the frozen protocol, with the timeout threshold recorded per task.
• Recovery attempts are counted: mean recovery attempts per successful trial and per failed trial, plus the recovery success rate. A policy that "succeeds" after six grasp retries has a different operational cost than one that succeeds on the first attempt, and deployment planning needs to know which one it is buying.
• If the budget changes between runs, the protocol version changes, and old scores lose comparability — the versioning discipline from the benchmark gaming post applies to time budgets with the same force as to environments.