Bench2Drive: every catch is a vehicle collision

Across 27 routes and 10 driver faults, seven faults are caught by no route, one route catches everything the others do, and all 72 catches went through collision_vehicle.

Published: 28 September 2026
Corpus: Bench2Drive (28 routes)
Kept by: Shanghai Jiao Tong University, Thinklab

AccidentTwoWays_25845 at seed 1, re-run with a chase camera: clean, PDM-Lite passes; with the occlude-target fault (nearest vehicle or walker never reported) it fails through collision_vehicle - the check every one of the campaign's 72 catches went through.

Re-run for this post, seed 1 · clean: passed, as the campaign recorded · occlude-target 1: failed through collision_vehicle, as the campaign recorded

Bench2Drive is 220 short CARLA routes, five for each of 44 scenario types, scored by the CARLA Leaderboard's infraction list - the closed-loop benchmark most end-to-end driving papers now report. We asked of it: if the driver got worse, would any route fail?

What we ran

Routes	28: one per scenario type, the first outside CARLA's two large towns - every type that has one
Driver	PDM-Lite, carla_garage's rule-based expert (203 of 220 routes a success by Bench2Drive's rule), run unmodified
Pass rule	Bench2Drive's own: the route completes with no infraction other than minimum speed; each infraction a named check
Faults	on the actor list PDM-Lite plans with - a sight limit, stale poses, the list emptied for 5 s, the nearest actor hidden - and on its steering, throttle and brake
Runs	1,574 on one rented CPU pod, rendering nothing, over 8 hours

We chose the strongest open driver on purpose: a route that cannot tell a degraded PDM-Lite from a sound one is a route that cannot tell much.

Three faults are caught, and 20 routes each catch all three

Every catch is one of: nothing visible past 5 m, the actor list emptied for five seconds, or the nearest vehicle or walker never reported. 20 of the 27 scored routes catch exactly those three, so any one of them catches everything the other 26 catch. AccidentTwoWays_25845 is the one the corpus analysis keeps; it is not the only one that would do. Judged one at a time, 23 routes are duplicates under this catalogue - which is not the same as useless.

Seven faults, given to 25-26 routes each, caught by none

0.3 s and 0.6 s of lag on every vehicle and walker's pose
nothing visible past 25 m
steering, throttle and brake rounded to a grid of 0.25 and of 0.5
noise of 0.1 and 0.3 on steering and throttle

Every one of those runs was delivered - the fault changed what PDM-Lite saw or commanded on 200 to 400 ticks - and every route finished with no infraction Bench2Drive counts. Outside the campaign, a whole second of lag and the coarsest possible controls also passed; noise at full strength finally failed, through leaving the lanes, which is how we know the control faults reach the car at all.

Every catch is a vehicle collision

All 72 catches went through collision_vehicle - including on the three pedestrian scenarios. collision_pedestrian, collision_layout and red_light never failed under any fault, though 24 to 26 routes each caught a fault aimed at those hazards. We have not looked into why the pedestrian check never fires: the records carry the check that failed, not what the car hit first.

One more thing about what is counted: a minimum-speed infraction was recorded on every run that finished, clean ones included, and Bench2Drive's rule never counts it. The nine that did not finish were never scored: eight whose CARLA server did not come up, and one that hung until it ran out of time.

The routes at the edges

ParkedObstacle_24757 caught none of its ten faults, all decided. carla_garage's own record for it has a vehicle collision; three clean runs here did not reproduce one.
OppositeVehicleRunningRedLight_26944 is the only route observed to fail by getting stuck and running out of time - under the 5 m sight limit.
MergerIntoSlowTrafficV2_26394 was set aside: PDM-Lite fails it with nothing injected, and carla_garage's own record shows the same infraction.

What this does not show

The 16 scenario types that exist only in CARLA's two large towns, and four of the five routes of every type measured.
One seed.
Any learned driver. PDM-Lite also plans on scenario metadata and a traffic-light list no fault here touches; a camera driver would be degraded differently, and a route that cannot tell a degraded PDM-Lite apart may well tell a degraded UniAD.

Check it yourself

Every simulator run behind this post is published: what ran, under which fault, and whether it passed. Replaying a recording runs no simulator - the same binary reads the runs back and prints the report the figures came from. In the browser it runs in a sealed tab that cannot send anything anywhere; with the CLI it is one command.

Bench2Drive with PDM-Lite

1,574 recorded runs of 28 CARLA routes, replayed into the report the post's figures came from. Its totals include 72 caught, 1,574 simulator runs, 28 scenarios - the figures quoted above.

Replay it in your browser runs.jsonl suite.json catalogue.json

mutational stale --replay bench2drive.runs.jsonl \
    --suite bench2drive.suite.json \
    --catalogue bench2drive.catalogue.json \
    --fingerprint pdm-lite@carla_garage-f22bc49 --baseline 3 --trials 5 --seed 1 --curtail=false --quiet

About these figures. Every number here comes from one sweep, recorded run by run, and each verdict is relative to one system under test and one fault catalogue - never a statement about the world. We have not yet discussed these findings with the people who keep Shanghai Jiao Tong University, Thinklab. If you do, and something here is wrong or unfair, write to admin@mutational.in and we will correct it in place. Every run behind it is published above, and replays into these figures.