What four open simulation corpora cannot catch
We made the system under test worse, on purpose, in NAVSIM, LIBERO, ArduPilot's SITL suite and Bench2Drive, and asked each scenario whether it noticed. The same four gaps showed up in all four.
- Published
- Corpus
- NAVSIM, LIBERO, ArduPilot, Bench2Drive
- Kept by
- Tübingen, UT Austin, ArduPilot, SJTU
A scenario that passes and could not have failed is green in CI and tests nothing. The only way to tell it apart from one that is doing its job is to make the system worse on purpose and see whether the scenario objects. That is all Mutational does: it injects a fault from a catalogue - a blinded sensor, a stale picture, a weaker grip - re-runs each scenario, and reads the scenario's own pass/fail checks.
We pointed it at four open corpora that other people keep and publish, and asked each the same question: if the system under test got worse, would any scenario fail?
| Corpus | Kept by | System under test | Runs |
|---|---|---|---|
| NAVSIM navtest (150 scenes) | University of Tübingen | PDM-Closed | 9,097 |
| LIBERO (40 tasks) | UT Austin | pi0.5 (Physical Intelligence) | 3,856 |
| ArduPilot SITL autotest (32 tests) | ArduPilot | ArduPlane, ArduCopter | 1,369 |
| Bench2Drive (28 routes) | SJTU Thinklab | PDM-Lite | 1,574 |
Every campaign first proved its own harness: a canary that removes the system's whole observation was caught, so "caught nothing" below is a measurement and never a harness that injected nothing. The whole thing cost $4.69 of rented compute.
The same four gaps, in all four
1. The scenes exist, and the assertions do not
Catches funnel through one or two checks. The checks that would say what went wrong rarely or never fire.
- Bench2Drive: all 72 catches went through
collision_vehicle. The pedestrian, layout and red-light checks never failed under any fault. - LIBERO: every catch is a goal predicate. Nothing asserts that an object was not dropped, knocked over or picked up by mistake.
- ArduPilot: 84 of 136 catches (62%) are the vehicle refusing to arm - the fault kept it on the ground, and no in-flight check was needed.
- NAVSIM is the exception: all eight of its checks fail through some fault.
2. Degraded perception often goes unnoticed
- NAVSIM: a planner that cannot see past 50 m is caught by 0 of 118 scenes.
- LIBERO: with the scene camera black for the whole episode, pi0.5 is caught in 11 of 36 tasks.
- Bench2Drive: a 25 m sight limit is caught by no route.
- ArduPilot: 370 ms of GPS lag is caught by no test.
3. Most of each corpus is redundant
| Corpus | Fewest scenarios that catch, between them, everything the corpus catches |
|---|---|
| Bench2Drive | 1 of 27 |
| ArduPilot | 3 of 31 |
| NAVSIM | 4 of 123 |
| LIBERO | 4 of 36 |
In Bench2Drive the one could be any of 20 routes, each of which catches all of it. In the other three no single scenario does; it takes the three or four together. That is relative to one fault catalogue, and never an argument to delete anything: a wider catalogue splits some of these apart.
4. Scoring rules can hide a failure
- NAVSIM's human-penalty filter forgave red lights a planner with no traffic-light input ran, in 8 scenes.
- LIBERO's evaluation stops at the first instant the goal is true, and looks at nothing afterwards.
- Bench2Drive records a minimum-speed infraction on every run that finished, clean ones included, and never counts it.
The posts
- NAVSIM: ten scenes a parked car passes
- LIBERO: the scene camera most tasks do not need
- ArduPilot: finding TerrainRally without being told
- Bench2Drive: every catch is a vehicle collision
What this does not show
- Each corpus was sampled - 150 of 12,146 NAVSIM scenes, 32 of about 750 ArduPilot tests, 28 of 220 Bench2Drive routes - and swept at one seed.
- Each verdict is about one system: PDM-Closed and PDM-Lite are privileged planners, and a learned end-to-end driver would be degraded differently.
- The faults are ours, stated in each post. "Caught by nothing" is about this catalogue.
- Nothing here measures how well any of these systems drives or grasps. It measures what the tests can tell apart.
Check it yourself
Every simulator run behind these figures is published: what ran, under which fault, and whether it passed. Replaying a recording runs no simulator - the same binary reads the runs back and prints the report the figures came from. In the browser it runs in a sealed tab that cannot send anything anywhere; with the CLI it is one command.
NAVSIM navtest, PDM-Closed
9,097 recorded runs of 150 navtest scenes, replayed into the report the post's figures came from.
Replay it in your browserruns.jsonlsuite.jsoncatalogue.json
mutational stale --replay navsim-pdm-closed.runs.jsonl \
--suite navsim-pdm-closed.suite.json \
--catalogue navsim-pdm-closed.catalogue.json \
--fingerprint pdm-closed --baseline 3 --trials 5 --seed 1 --curtail=false --quietLIBERO with pi0.5
3,856 recorded runs of 40 LIBERO tasks, replayed into the report the post's figures came from.
Replay it in your browserruns.jsonlsuite.jsoncatalogue.json
mutational stale --replay libero.runs.jsonl \
--suite libero.suite.json \
--catalogue libero.catalogue.json \
--fingerprint pi05_libero --baseline 3 --trials 5 --seed 1 --curtail=false --quietArduPilot SITL autotest at 89fdf67824
1,369 recorded runs of 32 Plane and Copter autotests, replayed into the report the post's figures came from.
Replay it in your browserruns.jsonlsuite.jsoncatalogue.json
mutational stale --replay ardupilot.runs.jsonl \
--suite ardupilot.suite.json \
--catalogue ardupilot.catalogue.json \
--fingerprint ardupilot@89fdf67824 --baseline 5 --trials 3 --seed 1 --curtail=false --quietBench2Drive with PDM-Lite
1,574 recorded runs of 28 CARLA routes, replayed into the report the post's figures came from.
Replay it in your browserruns.jsonlsuite.jsoncatalogue.json
mutational stale --replay bench2drive.runs.jsonl \
--suite bench2drive.suite.json \
--catalogue bench2drive.catalogue.json \
--fingerprint pdm-lite@carla_garage-f22bc49 --baseline 3 --trials 5 --seed 1 --curtail=false --quietAbout these figures. Every number here comes from one sweep, recorded run by run, and each verdict is relative to one system under test and one fault catalogue - never a statement about the world. We have not yet discussed these findings with the people who keep Tübingen, UT Austin, ArduPilot, SJTU. If you do, and something here is wrong or unfair, write to admin@mutational.in and we will correct it in place. Every run behind it is published above, and replays into these figures.