Domain randomization vs adversarial stress testing

Domain randomization and adversarial stress testing sound like the same idea: vary the physics, see if the policy survives. They are different categories of work, and conflating them is how robustness claims get inflated.

One is a training technique. The other is an evaluation instrument. Using one to certify the other is a category error with real consequences.

What domain randomization actually does

Domain randomization (DR) samples environment parameters during training — mass, friction, lighting, sensor noise — from a declared distribution. The policy learns to operate across that distribution. It is one of the most successful ideas in sim-to-real transfer, and its value is training-side: it expands the set of environments the policy has seen.

The critical property of DR is that the sampling distribution is chosen by the trainer. That makes it a bet, not a test:

• If the real-world parameter falls inside the sampled range, the policy may transfer well.
• If it falls outside — heavier payload, wetter surface, degraded sensor — nothing in DR guarantees graceful degradation.
• If the sampling distribution is narrow, DR mainly improves the training curriculum; it says nothing about the robustness boundary.

A policy trained with DR can be robust within its training distribution and still fail catastrophically just past the edge. Nothing about DR reveals where that edge is.

What stress testing actually measures

Adversarial stress testing flips the sampling logic: instead of the trainer choosing the distribution, the evaluation chooses worst-case conditions — tiered perturbations of mass, friction, contact properties, and sensor noise, applied in increasing severity — and records where the policy degrades.

The output is a degradation curve, not a training signal. It answers the question DR cannot: how far past nominal can the policy go before it fails, and how does it fail?

This is the design behind the tiered perturbations in Haga's policy pillar: Lift, Stack, PickPlaceCan, and Door are stressed through severity tiers with defined thresholds, Wilson intervals, and logged failure cases. See the Lab for the published curves — nominal success degrading to near-zero at high severity, with failure modes named per tier.

The category error

The common mistake is reporting DR training coverage as evidence of robustness:

"We used domain randomization, therefore our policy is robust."

The correct statement is: "We trained with DR over ranges [mass ±30%, friction ±0.2, lighting variance X]. Stress testing showed the policy holds through severity tier 2 and degrades predictably at tier 3, with slip failures dominating."

One is a statement about training. The other is a statement about behavior under stress. Robustness claims need the second. A sim-to-real benchmark built on DR coverage alone is a leaderboard of the training recipe, not a measure of transfer.

When the two work together

The right workflow treats them as complements:

1. Train with DR to widen the policy's operating envelope.
2. Freeze the protocol — seeds, perturbation tiers, thresholds — before evaluation.
3. Stress-test with an independent checker that was not involved in training.
4. Report the degradation curve with failure taxonomies, not a single pass rate.

The independence matters for the same reason audits matter in finance: the team that trained the policy also chose the DR ranges, and self-certification of robustness is exactly the kind of claim that should be checked by someone who did not benefit from it being true. That is the logic behind independent verification for physical AI.

The one-sentence summary

Domain randomization is how you train a policy to survive a distribution. Stress testing is how you find out which distribution it actually survives. Do not let the first sentence be cited as evidence for the second.

A worked example: the lifted box

To see the difference concretely, take a pick-and-lift policy trained with DR over payload mass in [0.5 kg, 1.0 kg] and friction in [0.4, 0.8].

• The DR claim is: "trained across those ranges." True, and it is exactly what a trainer would say.
• The stress-test finding is something else entirely: nominal performance is strong, but at 1.5 kg — 50% past the trained envelope — the policy's success collapses, and the dominant failure mode is slip at lift-off, not grasp, not trajectory. The degradation is graceful (monotonic, predictable) up to 1.3 kg, then cliff-like.

These are two different facts about the same policy. The first describes its education; the second describes its limits. A procurement decision, a pilot scope, or a safety review is decided by the second fact — and only the stress test produces it.

This is the design behind the tiered perturbations in Haga's policy pillar: Lift, Stack, PickPlaceCan, and Door are stressed through severity tiers with defined thresholds, Wilson intervals, and logged failure cases. See the Lab for the published curves — nominal success degrading at high severity, with failure modes named per tier.

Common DR misdesigns that inflate robustness claims

Even teams that do both training and testing often make the evaluation weaker than it looks:

• Reporting the DR range as the robustness envelope. "Robust over [0.5, 1.0] kg" is only true if the policy was actually tested across that range — sampled verification across the range, not just trained on it. Training coverage does not equal tested coverage.
• Testing inside the training distribution. A stress test that stays within the DR-sampled range is a regression test, not a stress test. The interesting region is past the envelope edge, where the policy has never seen a parameter combination.
• Correlated perturbations. Varying mass and friction together in the same direction masks which parameter drives the failure. Perturb one axis per tier, then combine at the boundaries.
• Reporting the best seed. Every policy has a seed lottery. Report the median and the spread across seeds, with the failure distribution — this is the same discipline as manipulation metrics that survive a skeptic, and it applies to every perturbation tier.
• No control condition. A nominal-tier run must anchor every stress run. Without the nominal baseline and its interval, "degradation at tier 3" is uninterpretable — you cannot tell whether the stress caused it or the policy was never good at that task configuration.

The pattern to internalize: DR decides what the policy might have learned; stress testing decides what the policy did learn, and where it breaks. Teams that conflate the two produce robustness claims that dissolve on first contact with an out-of-distribution deployment site.