Pillar 1 · Policy verification
ShippedStack stress
Two-cube stack — same gate, harder placement under stress.
Three severity tiers of mass/friction stress on cubeA in robosuite Stack, paired seeds, Wilson confidence intervals. Mild gate PASSes; severe stress collapses success as the cube slips after grasp during lift/stack. The curve — not the mild badge — is the diligence artifact.
What we tested
Where does a Stack policy break as cubeA mass and friction move further from nominal?
- success rate, nominal → severe
0.96 → 0.20
success rate, nominal → severe
- severe episodes: grasp acquired, then lost
80%
severe episodes: grasp acquired, then lost
- gate tier (mild) at defined thresholds
PASS
gate tier (mild) at defined thresholds
Charts
Success vs grasp-failure across severity
rate
Mean peak end-effector force
N (contact proxy)
Metrics
- gate: stress success ≥ 80% of nominal
- grasp-fail Δ ≤ 15 pp
- Wilson 95% CIs on all rates
| Metric | nominal | mild | moderate | severe |
|---|---|---|---|---|
| Success rate | 0.96 | 0.82 | 0.72 | 0.20 |
| Success 95% CI | [0.87, 0.99] | [0.69, 0.90] | [0.58, 0.83] | [0.11, 0.33] |
| Grasp-failure rate | 0.04 | 0.18 | 0.28 | 0.80 |
| Mean peak EE force (contact proxy) | 7.44 | 8.35 | 12.98 | 13.58 |
| Score (tier / nominal) | — | 0.85 | 0.75 | 0.21 |
| Verdict at gate thresholds | — | PASS | FAIL | FAIL |
Methodology
- Nominal baseline. Run Stack with default cubeA mass/friction; record success, grasp-fail, and peak EE force over paired seeds.
- Tiered stress. Resample cubeA mass and sliding friction into mild → moderate → severe ranges while holding seeds paired with the baseline.
- Gate + degradation. Score gate tier against success-ratio and grasp-fail Δ thresholds; report severe/nominal as the degradation headline.
- Failure logging. Log each failed episode with sampled physics; classify grasp loss with a 5-step debounce after acquisition.
Seeds: 0..49 paired across conditions (50 episodes × 4 conditions)
- gate: stress success ≥ 80% of nominal
- grasp-fail Δ ≤ 15 pp
- Wilson 95% CIs on all rates
Failure case · Grasp slip
The cube is gripped — then slips before stack credit. That is the finding.
A Stack policy that scores 0.96 at nominal looks fine until cubeA mass rises and sliding friction falls. Without severity tiers and an honest grasp-loss metric, this would have stayed a silent green PASS.
How: Paired seeds across mild → moderate → severe mass/friction ranges. Grasp failure = never held, or grasp lost before stack credit (5-step debounce). Each failed episode is logged with the exact sampled physics.
80%
of severe episodes: grasp acquired, then lost (slip)
| Seed | Mass (kg) | Sliding μ | Mode |
|---|---|---|---|
| 0 | 2.37 | 0.082 | grasp lost (slip) |
| 1 | 1.13 | 0.168 | grasp lost (slip) |
| 2 | 0.90 | 0.086 | grasp lost (slip) |
| 3 | 2.47 | 0.282 | grasp lost (slip) |
Same grasp-slip taxonomy as Lift/PickPlaceCan. Mild gate PASS is the entry bar; severe degradation is the story.
Notes
Sim-only Stack stress. Grasp/place failure under mass/friction tiers — not a substitute for real-hardware validation.
Sim-only — validated concept evidence, not a substitute for real-hardware validation or the full platform vision.