Findings
What open corpora catch, and what they cannot
We ran Mutational against simulation suites other people keep and publish. It makes the system under test worse on purpose - a fault from a catalogue - and asks each scenario's own checks whether they notice.
- · NAVSIM, LIBERO, ArduPilot, Bench2Drive
What four open simulation corpora cannot catch
We made the system under test worse, on purpose, in NAVSIM, LIBERO, ArduPilot's SITL suite and Bench2Drive, and asked each scenario whether it noticed. The same four gaps showed up in all four.
Read it - · NAVSIM navtest
NAVSIM: ten scenes a parked car passes
On 150 navtest scenes, a planner that cannot see past 50 m is never caught, 10 scenes pass with the car standing still, and the human-penalty filter forgives red lights a degraded planner runs.
Read it - · LIBERO (Spatial, Object, Goal, 10)
LIBERO: the scene camera most tasks do not need
With pi0.5, losing the wrist camera is caught in 35 of 36 tasks. Losing the whole scene camera is caught in 11.
Read it - · ArduPilot SITL autotest
ArduPilot: finding TerrainRally without being told
ArduPilot's own issue #33740 says Plane.TerrainRally tests nothing. A sweep of 32 autotests, not told that, never saw its terrain assertion fail in 108 faulted runs.
Read it - · Bench2Drive (28 routes)
Bench2Drive: every catch is a vehicle collision
Across 27 routes and 10 driver faults, seven faults are caught by no route, one route catches everything the others do, and all 72 catches went through collision_vehicle.
Read it