ArduPilot: finding TerrainRally without being told
ArduPilot's own issue #33740 says Plane.TerrainRally tests nothing. A sweep of 32 autotests, not told that, never saw its terrain assertion fail in 108 faulted runs.
- Published
- Corpus
- ArduPilot SITL autotest
- Kept by
- The ArduPilot project
Two SITL flights of Plane.TerrainRally at ArduPilot 89fdf67824, one clean and one with 370 ms of GPS lag: the test fails if the plane is under 50 m above the ground, and in both it is never lower than 91 m while that check is on.
Re-run for this post, seed 1 · clean: passed, as the campaign recorded · lag 0.3: passed, as the campaign recorded
ArduPilot keeps about 750 simulation tests in CI, and a dictionary in the code of the ones it has switched off. In July 2026 one entry read "TerrainRally": "Passes vacuously due to helper alt-frame bugs", and issue #33740 put it plainly: the test "doesn't really test anything right now ... we're climbing away like crazy and merely asserting it remains above 50m AGL at all times."
Someone found that by hand. We wanted to know whether a sweep finds it without being told - and what else is in the suite like it.
What we ran
| Commit | 89fdf67824, the last one at which TerrainRally was in CI |
|---|---|
| Tests | 32 Plane and Copter flight tests, autotest unmodified, one Docker container per run |
| Faults | ArduPilot's own SIM_ parameters: GPS lag, GPS byte loss, GPS altitude drift, loop-time jitter, IMU noise, reduced thrust |
| Runs | 1,369 on one laptop, in 35 minutes |
TerrainRally, found - mostly
| Faulted runs that failed through its terrain assertion | 0 of 108, across three sweeps |
|---|---|
| What it does catch | 3 of 12 faults, all before terrain following starts: GPS byte loss stops it arming, and 40% thrust stops it climbing to take-off height |
| Deletion triage | DELETE: all three are caught elsewhere, by Copter.BrakeZ alone |
| Height above terrain while its 50 m check is installed | 92 to 758 m, measured in two clean runs |
The plane spends the test hundreds of metres above the floor it is checking, so no fault brings it within reach of its own assertion. That is the issue's account, arrived at from the other side.
We want to be exact about how far this went. The sweep does not call TerrainRally inert, because it catches three faults - just never through its own assertion. And our worklist, with the fault catalogue as we shipped it, lists the terrain hazard as "unconfirmed" rather than "assertion silent", because one other test caught a fault aimed at it through a free-form assertion our catalogue does not map. Mapping that one line differently, with no new runs, names TerrainRally as the silent assertion. The observation - its assertion never failed - does not change; what the report calls it depends on a mapping choice.
The rest of the suite
- 84 of 136 catches (62%) are the vehicle refusing to fly - never ready to arm, or its estimator never healthy. Only 52 happen in the air.
- Two faults are caught by nothing: IMU noise at the suite's own "noisy" level, given to 28 tests; and 370 ms of GPS lag, given to 27.
- One test alone notices GPS altitude drift (Copter.WatchAlts); one alone notices doubled IMU noise (Copter.CircleSpeed).
- Three Copter tests - WatchAlts, CircleSpeed, RTLSpeed - catch everything the 32 catch between them.
- Copter.AltEstimation caught none of its 12 faults.
- Mode changes: 22 tests wait on one, and none failed through it; 28 caught a fault aimed at that hazard, all through another check.
Two tests depend on the tests before them
Run on its own, TerrainRally stalls: its first act floods the link with terrain requests when the vehicle's terrain cache is empty, and autotest's watchdog fires. In CI it inherited a warm cache from about 80 earlier tests. We swept it from the cache Plane.TerrainLoiter leaves behind, with its code unchanged, and then it passed every clean run. Plane.DO_CHANGE_ALTITUDE fails every clean run on its own at this commit; ArduPilot added a terrain handler to it a month later.
What this does not show
- 32 of about 750 tests, one commit, one seed. Rover, QuadPlane and Sub were not run.
- Which hazard each fault is aimed at is our claim; only the check that failed is an observation.
- A test that catches nothing here may catch a fault outside this catalogue.
Check it yourself
Every simulator run behind this post is published: what ran, under which fault, and whether it passed. Replaying a recording runs no simulator - the same binary reads the runs back and prints the report the figures came from. In the browser it runs in a sealed tab that cannot send anything anywhere; with the CLI it is one command.
ArduPilot SITL autotest at 89fdf67824
1,369 recorded runs of 32 Plane and Copter autotests, replayed into the report the post's figures came from. Its totals include 136 caught, 1,369 simulator runs, 32 scenarios - the figures quoted above.
Replay it in your browserruns.jsonlsuite.jsoncatalogue.json
mutational stale --replay ardupilot.runs.jsonl \
--suite ardupilot.suite.json \
--catalogue ardupilot.catalogue.json \
--fingerprint ardupilot@89fdf67824 --baseline 5 --trials 3 --seed 1 --curtail=false --quietAbout these figures. Every number here comes from one sweep, recorded run by run, and each verdict is relative to one system under test and one fault catalogue - never a statement about the world. We have not yet discussed these findings with the people who keep The ArduPilot project. If you do, and something here is wrong or unfair, write to admin@mutational.in and we will correct it in place. Every run behind it is published above, and replays into these figures.