Benchmark gaming in robot learning: when leaderboards lie

There is a rule of benchmarking that never makes it into the paper: every benchmark that gates funding or attention eventually gets gamed. Not because researchers are dishonest — because the incentives line up. If a number determines a release decision, a funding round, or a headline, optimizing the number becomes part of the task.

Robotics is not exempt. It may be the most gameable field of all, because the evaluation surface is enormous and the protocols are young.

The four failure modes

1. Leaderboard overfitting

When a fixed benchmark persists, teams tune to it. Hyperparameters, reward shaping, and even environment tweaks drift toward the specific test conditions. The benchmark stops measuring general capability and starts measuring benchmark-specific memorization.

This is the RL analog of test-set contamination in supervised learning — and it is worse, because the "test set" (the simulator) is cheap to query a million times. The policy can be optimized directly against the evaluation distribution without any explicit cheating. Every team on a leaderboard claims they never touched the test environment. The leaderboard cannot tell you who is right.

2. Selective reporting

The benchmark score is the best of N runs, or the best seed, or the best checkpoint. Failure modes are reported as "transient." The n is small and the interval is missing. None of this is falsified — it is just the selection of evidence that survives legal review.

A score without a protocol version, seeds, or intervals is a press release, not a measurement. Ask for the failure distribution and watch how many teams produce it.

3. Protocol drift

Benchmark maintainers "fix" the environment between versions: new physics constants, changed reward scaling, reordered episodes. Version ambiguity makes old scores non-comparable and new scores non-reproducible. Without pinned seeds and versioned environments, the leaderboard is a moving target that everyone can cite in their favor.

4. Evaluation proxy substitution

The most subtle failure mode: the reported metric is a proxy for what you actually care about. Success rate on a clean map substitutes for deployment robustness; FVD substitutes for physical consistency; a demo video substitutes for a cohort. The proxy is real and measurable — it just answers the wrong question.

Why robotics is uniquely exposed

Robotics has a hard realism ceiling on automated evaluation: physical setups are expensive, so n is tiny; simulators are cheap, so overfitting is easy; and demos are cinematic, so selective reporting travels well. Every weakness compounds. The field's leaderboards are simultaneously the most important and the least trustworthy evidence in embodied AI.

What keeps numbers honest

Honest benchmarks share a protocol shape, not a particular metric:

1. Frozen protocols. Seeds, perturbation sets, scoring scripts, and thresholds are versioned and public before the cohort runs.
2. Failure taxonomies. Success-only tables hide the science. Slip vs timeout vs contact explosion are different bugs; report them separately.
3. Intervals that travel with the mean. Wilson intervals make small-n noise visible.
4. Held-out cohorts. The evaluation distribution should be locked before the final tuning pass — see the Physics-IQ held-out cohort as a worked example of what a frozen protocol with claim boundaries looks like.
5. Independent scoring. The team that trained the policy should not be the only team that scored it. This is the core argument for independent verification: the checker must not benefit from the claim being true.

Reading a leaderboard like a skeptic

Before citing any robotics leaderboard, ask five questions: What protocol version produced the number? Which seeds? What was held out versus rehearsed? Where is the failure distribution? Who scored it?

Teams that answer all five are rare — which is precisely why the answers are the signal. A leaderboard that cannot produce them is not evidence; it is marketing with a ranking. The benchmark design guide covers how to build the honest version from scratch.

Worked example: the crowded table

Take a common scene in manipulation papers: a pick-and-place benchmark with a table of objects, a score of "90% task success," and a demo video. Run the five questions and the number disintegrates in a specific order:

1. Protocol version — the team used the v2 environment with a custom reward wrapper that was never released. The v1-to-v2 diff changed contact parameters, so the published score is not reproducible even on the "same" benchmark.
2. Seeds — the score is the best of 5 seeds, and the seed-to-seed spread is 61% to 90%. Reporting the max turns a fragile policy into a headline.
3. Held-out vs rehearsed — the test condition is a subset of the training distribution: the same object set, same spawn ranges, same table. There is no held-out condition with novel geometry or perturbation.
4. Failure distribution — of the 10% failures, seven were timeouts from grasp retries and three were placement misalignments. "90% success" hides that the recovery loop, not the grasp, is the weakest subsystem.
5. Who scored — the team wrote the scoring script and ran the runs. No independent party re-scored, and the raw episode logs were never released.

None of this required dishonesty. Each step is a rational choice under incentive pressure — and together they make the number meaningless as a capability claim. This is why independent verification is the structural fix, not a request for better manners.

What honest leaderboards look like

The benchmark landscape is not doomed to be a game-theoretic race to the bottom. A leaderboard that resists gaming has three design features:

• A permanent holdout. A portion of the evaluation distribution is frozen and physically/algorithmically withheld from everyone — released only as scoring runs, never as training data. The held-out cohort design from Physics-IQ shows the mechanics: the cohort and thresholds are locked before scoring begins.
• Mandatory artifacts. Ranking requires submitting raw episode logs, seeds, and the scoring invocation. Without artifacts, there is no entry. Artifacts turn "trust me" into "check me."
• Independent scoring. The benchmark operator re-runs submissions in their own sandbox rather than trusting submitted scores. This is standard practice in NLP (hidden test splits) and almost absent in robotics.

The pattern is not new — it is the pattern of every credibility market that matured: financial audits, compiler conformance suites, chess engine tournaments. Robotics benchmarks are at the stage where the field still believes the scoreboard. The teams that treat their benchmarks as audit instruments will be the ones whose numbers survive contact with diligence.