NAVSIM: ten scenes a parked car passes

On 150 navtest scenes, a planner that cannot see past 50 m is never caught, 10 scenes pass with the car standing still, and the human-penalty filter forgives red lights a degraded planner runs.

Published: 28 September 2026
Corpus: NAVSIM navtest
Kept by: University of Tübingen, Autonomous Vision Group

On navtest scene 61a9157ac995542c, PDM-Closed moves 4.9 m and a car told to hold where it is moves 0.3 m, and NAVSIM's scorer passes both on every check; with nothing beyond 50 m in its input, PDM-Closed plans the same path and passes too.

Re-run for this post, seed 1 · clean: passed, as the campaign recorded · canary (drop): passed, as the campaign recorded · blind 0.5: passed, as the campaign recorded

NAVSIM scores a planner on real nuPlan scenes: 1.5 s of history, one 4 s trajectory, rolled out against the logged future and scored as EPDMS - four multiplicative gates times a weighted mean of subscores. We asked of it what we ask of any corpus: if the planner got worse, would the scene notice?

What we ran

Scenes	150 of navtest's 12,146, a seeded sample - a sample of navtest, not navtest
Planner	PDM-Closed, configured as NAVSIM builds its reference; and NAVSIM's constant-velocity agent
Scoring	NAVSIM's own `pdm_score`, human-penalty filter on, as NAVSIM ships it
Faults	10 kinds, 26 settings: sight limits, pose lag, drift and yaw error, dropped traffic lights, output noise, a phantom car
Runs	9,097 for PDM-Closed, on a laptop, CPU only, in under 16 minutes

NAVSIM does not decide pass or fail, so we did it in one stated rule: each gate below 1 is a failed check under its own name - at_fault_collision, off_drivable_area, wrong_way, ran_red_light - as is a zero in TTC, lane keeping or comfort. One threshold is ours, not NAVSIM's: ego progress below 0.5 is low_progress.

Ten scenes pass with the car standing still

The first thing every sweep runs is a canary: the planner is given nothing and the car is told to hold where it is. 10 of the 123 scored scenes pass every check under it - three of them starting at 4.5 to 5.6 m/s. A scene in which stopping is a full pass cannot tell a planner that stops from one that drives.

A planner that cannot see past 50 m is never caught

Sight	Scenes that caught it	Survived	Nothing that far away
75 m	0	72	51
50 m	0	118	5
25 m	11	109	3

At 50 m, not one of 263 runs failed a single check. Part of that is physics - four seconds at urban speed does not reach an object 50 m away - and it is still the measurement: on this sample, EPDMS cannot tell a planner with 50 m of perception from one with 100 m.

Removing every traffic light is noticed in 6 of 61 scenes

With no traffic-light status given to the planner at all, 6 of the 61 scored scenes that have a light caught it; 55 passed every check, red-light compliance included. Across all 26 faults ran_red_light fired in 24 runs, the fewest of any check. The check works where a scene exercises it. Few do.

The human-penalty filter forgives a degraded planner

NAVSIM sets a metric back to passing when the logged human driver also failed it. With every light removed from the planner's input, ran_red_light failed and was forgiven in 8 scenes: in those, the red-light gate cannot fail for any planner. Clean, the filter forgave a failed check in 9 scenes and turned 6 of them into passes. We have not looked into why the logged driver fails that gate there.

23 scenes cannot tell PDM-Closed from driving straight ahead

NAVSIM's constant-velocity agent sees nothing but its own speed. It passes every check clean in 23 of the 150 scenes, and PDM-Closed passes all 23 as well: there, the benchmark gives both the same pass. It is a per-scene version of what Driving on Memory argues about the benchmark as a whole, and of why NAVSIM ships a blind ego-status baseline.

Most scenes duplicate each other

Judged one at a time against the rest, 99 of 150 scenes catch nothing another scene does not; 2 earn their place; 49 cannot be judged (27 fail clean, 22 left unsettled). Four scenes catch everything the 123 scored scenes catch between them. That is relative to this fault catalogue, and a budget observation - not a verdict on any scene.

What does get caught

Pose faults, most: a yaw error in 105 scenes, drift in 60, 1.5 s of lag in 59, mostly through off-road, lane and wrong-way checks. A phantom stopped car 5 m ahead is caught in 86, almost always through low_progress - our threshold. Output noise of 0.125 m and 0.25 m is caught nowhere; the scorer's tracker absorbs it. Unlike the other three corpora we measured, every one of NAVSIM's checks fails through some fault.

What this does not show

150 of 12,146 scenes, one seed, navtest only - navhard was not run.
Two privileged planners. No learned end-to-end model: those need the 223 GB sensor split.
The progress threshold (0.5) is ours. Everything else is NAVSIM's own semantics.
27 scenes were set aside because PDM-Closed fails them with nothing injected; in 24 of them NAVSIM's own reference trajectory fails through the same checks.

Check it yourself

Every simulator run behind these figures is published: what ran, under which fault, and whether it passed. Replaying a recording runs no simulator - the same binary reads the runs back and prints the report the figures came from. In the browser it runs in a sealed tab that cannot send anything anywhere; with the CLI it is one command.

NAVSIM navtest, PDM-Closed

9,097 recorded runs of 150 navtest scenes, replayed into the report the post's figures came from. Its totals include 9,097 simulator runs, 150 scenarios - the figures quoted above.

Replay it in your browser runs.jsonl suite.json catalogue.json

mutational stale --replay navsim-pdm-closed.runs.jsonl \
    --suite navsim-pdm-closed.suite.json \
    --catalogue navsim-pdm-closed.catalogue.json \
    --fingerprint pdm-closed --baseline 3 --trials 5 --seed 1 --curtail=false --quiet

NAVSIM navtest, constant velocity

2,382 recorded runs of NAVSIM's constant-velocity agent on the same 150 scenes.

Replay it in your browser runs.jsonl suite.json catalogue.json

mutational stale --replay navsim-constant-velocity.runs.jsonl \
    --suite navsim-constant-velocity.suite.json \
    --catalogue navsim-constant-velocity.catalogue.json \
    --fingerprint constant-velocity --baseline 3 --trials 5 --seed 1 --curtail=false --quiet

About these figures. Every number here comes from one sweep, recorded run by run, and each verdict is relative to one system under test and one fault catalogue - never a statement about the world. We have not yet discussed these findings with the people who keep University of Tübingen, Autonomous Vision Group. If you do, and something here is wrong or unfair, write to admin@mutational.in and we will correct it in place. Every run behind it is published above, and replays into these figures.