Pick-and-place verification: beyond the one-video demo

Pick-and-place is the gateway drug of robot demos. One clean grasp, one clean drop, one confident caption. Factories, warehouses, and lab benches all care about variants of this skill — which is exactly why a single video is such a dangerous substitute for verification.

This guide is about turning pick-and-place from a demo into diligence.

What a demo proves (and only that)

A successful rollout proves that at least one trajectory worked under the conditions that appeared on camera. It does not prove:

• stability under object mass changes,
• tolerance to friction resampling,
• grasp integrity after lift,
• placement precision under contact noise,
• or behavior outside the rehearsed SKU set.

If your acceptance meeting ends when someone says “it worked on the video,” you are accepting anecdote as certification.

Split the task into measurable parts

Pick-and-place looks atomic. Evaluation should not.

Motion success

Did the arm reach pre-grasp, lift height, and place pose within tolerances? This can pass while the object is already slipping.

Grasp integrity

Did the grasp remain stable through lift and transport? Track slip events, regrasps, and drops separately from end-effector pose error.

Place stability

Did the object remain in the target region after release — including after a short settle window? “Released above the bin” is not “placed.”

Recovery behavior

If the first grasp fails, does recovery terminate cleanly or thrash until timeout? Timeouts are failures; label them.

Reporting a single “success rate” that mixes these modes hides the subsystem you should fix.

Physics stress is the point of sim-first eval

When hardware time is scarce, simulation is how you multiply contact conditions. The mistake is using sim only to farm easy wins.

Haga’s policy pillar adversarially varies mass and friction on robosuite tasks including PickPlaceCan, alongside Lift, Stack, and Door. We report success rates with Wilson confidence intervals and show severe-tier failures rather than a binary PASS. Inspect lift stress results for the public shape of that evidence and read the methodology.

For pick-and-place specifically, borrow this structure even if your stack is not robosuite:

1. Mild / moderate / severe tiers for mass and friction.
2. Paired seeds across tiers so comparisons are not lottery tickets.
3. Per-mode failure counts (missed grasp, slip, place miss, timeout).
4. Held-out object geometries that did not appear in training montages.
5. Protocol versioning so last month’s plot remains interpretable.

Why Wilson intervals belong in your slide

At the sample sizes most labs actually run, a raw percentage lies with confidence. Wilson intervals make small-n uncertainty visible — the same discipline we use on generative cohorts where a held-out flag rate of 1.000 still carries a wide interval at n=9.

If you cannot show an interval, show the n. If you cannot show the n, you are not ready for external claims.

A practical protocol template

Use this as a starting checklist for an internal pick-and-place gate:

Field	Example
Protocol ID	pickplace-stress-v3
Task definition	Grasp can, lift above threshold, place in target XY with settle
Seeds	Fixed list, published with packet
Tiers	Mild / moderate / severe mass×friction
Primary metrics	Trial success, grasp-integrity rate, place-stability rate
Secondary metrics	Time-to-success, regrasp count, drop count
Failure taxonomy	Approach miss, slip, overload proxy, place miss, timeout
Non-claims	Not hardware certification; not cross-robot generalization

Share the packet with anyone who has to bet money on the next pilot.

When to escalate to hardware-in-the-loop

Sim stress is necessary and insufficient. Escalate when:

• severe tiers pass with stable failure modes (not flukes),
• the SKU set matches the cell you care about,
• sensing and gripper differences are acknowledged in the claim,
• and you are ready to reuse the same taxonomy on real trials.

Do not wait for perfection in sim to touch hardware — but do not let hardware anecdotes erase sim failures either. The Stanford AI Index (2026) reminder still applies: high sim success and low real-world household success can coexist in the same industry narrative.

World models in the pick-and-place loop

If your policy was trained with generative video, neural world models, or synthetic contact footage, add a world-model physics check. Visually plausible training clips can still freeze, hover, or break contact. Haga’s Physics-IQ-oriented work found real quiet controls at 0% flags and a CogVideoX held-out cohort at 100% flags via static_hover under protocol v1. Details: Physics-IQ lab.

Pick-and-place policies inherit the sins of their teachers.

What Haga will and will not claim

We will help you stress a pick-and-place policy under fixed physics protocols and produce shareable evidence with shown failures.

We will not declare your cell “production ready” from sim alone, run a public leaderboard vanity contest, or pretend one video is a benchmark.

Next step

If you have a policy checkpoint or a task definition ready for an independent run, request an eval. For the public posture, start at methodology and the lab.