mutational: 1 gated hazard(s) are unreachable by "homebody orchestration"; nothing below says anything about them ==================================================================== MUTATIONAL REPORT ==================================================================== runner exec: python3 -m mutational_homebody.runner {scenario} (recording) simulation runs 2205 in 10.136s settled early 10 inconclusive verdicts reached before their last trial, which could not have changed them, and 48 runs not spent. --curtail=false runs them discrimination 49.5% (52 of 105 injected defects detected) over the 105 defects decided, not the 164 planned: 27 defects have no verdict yet, so this samples the discrimination rather than measuring it scenarios 4 scored, 0 skipped, 16 inconclusive defects undeliverable 32 defects could not be expressed in the scenarios aimed at, and are excluded: not exposed, and not missed either. rarely delivered 11 defects landed on some placements and were refused on most, and ran out of attempts before there were trials enough to judge: retrieve_the_medicine occlude-target(orchestration, from 0s, x0.25) landed on 19 of 40 retrieve_the_medicine myopic(orchestration, from 0s, x0.25) landed on 6 of 40 retrieve_the_medicine myopic(orchestration, from 0s, x0.5) landed on 6 of 40 retrieve_the_medicine myopic(orchestration, from 0s, x0.75) landed on 18 of 40 retrieve_the_medicine reorder(orchestration, from 0s, x0.25) landed on 12 of 40 ... and 6 more No verdict, and not refused. More --trials buys more placements. ==================================================================== PER SCENARIO ==================================================================== retrieve_the_medicine 43.5% 10/23 flake 0% (23 of 41 planned) tidy_the_kitchen 44.8% 13/29 flake 0% (29 of 41 planned) tidy_the_kitchen_kept_as_found 51.7% 15/29 flake 0% (29 of 41 planned) retrieve_the_medicine_kept_as_found 58.3% 14/24 flake 0% (24 of 41 planned) ==================================================================== UNDETECTED DEFECTS ==================================================================== retrieve_the_medicine depth-error(orchestration, from 0s, x0.25) retrieve_the_medicine blind(orchestration, from 0s, x0.25) retrieve_the_medicine blind(orchestration, from 0s, x0.5) retrieve_the_medicine blind(orchestration, from 0s, x0.75) ... 9 more in this scenario tidy_the_kitchen phantom-actor(orchestration, from 0s, x0.25) tidy_the_kitchen occlude-target(orchestration, from 0s, x0.25) tidy_the_kitchen occlude-target(orchestration, from 0s, x0.5) tidy_the_kitchen depth-error(orchestration, from 0s, x0.25) ... 12 more in this scenario tidy_the_kitchen_kept_as_found phantom-actor(orchestration, from 0s, x0.25) tidy_the_kitchen_kept_as_found occlude-target(orchestration, from 0s, x0.25) tidy_the_kitchen_kept_as_found occlude-target(orchestration, from 0s, x0.5) tidy_the_kitchen_kept_as_found depth-error(orchestration, from 0s, x0.25) ... 10 more in this scenario retrieve_the_medicine_kept_as_found depth-error(orchestration, from 0s, x0.25) retrieve_the_medicine_kept_as_found blind(orchestration, from 0s, x0.25) retrieve_the_medicine_kept_as_found blind(orchestration, from 0s, x0.5) retrieve_the_medicine_kept_as_found blind(orchestration, from 0s, x0.75) ... 6 more in this scenario ==================================================================== WHAT THIS SUITE IS PAYING FOR ==================================================================== scenarios 4 scored defects caught 18 noticed by none 12 more, injected into several and caught in none scenarios needed 2 of 4 to catch all of them simulator runs 2205 this sweep ==================================================================== 12 DEFECTS WERE NOTICED BY NOTHING IN THIS SUITE ==================================================================== Each was injected into several scenarios, reached a verdict in all of them, and changed no outcome anywhere. Two things look exactly like this and this run cannot tell them apart: the suite shares a blind spot - a hole worth filling it cannot be caught here at all - worth taking out of the catalogue, not out of the suite Either way it sits in every scenario's denominator and none of them can score on it, so it lowers every figure above. blind(orchestration, from 0s, x0.25) 4 scenario(s), 4 of which catch other defects blind(orchestration, from 0s, x0.5) 4 scenario(s), 4 of which catch other defects blind(orchestration, from 0s, x0.75) 4 scenario(s), 4 of which catch other defects depth-error(orchestration, from 0s, x0.25) 4 scenario(s), 4 of which catch other defects sluggish(orchestration, from 0s, x0.25) 4 scenario(s), 4 of which catch other defects sluggish(orchestration, from 0s, x0.5) 4 scenario(s), 4 of which catch other defects stall(orchestration, from 0s, x0.25) 4 scenario(s), 4 of which catch other defects unsteady(orchestration, from 0s, x0.1) 4 scenario(s), 4 of which catch other defects unsteady(orchestration, from 0s, x0.3) 4 scenario(s), 4 of which catch other defects occlude-target(orchestration, from 0s, x0.25) 2 scenario(s), 2 of which catch other defects occlude-target(orchestration, from 0s, x0.5) 2 scenario(s), 2 of which catch other defects phantom-actor(orchestration, from 0s, x0.25) 2 scenario(s), 2 of which catch other defects ==================================================================== THE SET THAT COVERS EVERYTHING THIS SUITE COVERS ==================================================================== Ordered by what each adds that nothing before it did. Stop where the next scenario stops earning its place in the merge queue. 1. tidy_the_kitchen_kept_as_found +15 defect(s) depth-error(orchestration, from 0s, x0.5) drift(orchestration, from 0s, x0.25) drift(orchestration, from 0s, x0.5) drift(orchestration, from 0s, x0.75) early-grasp(orchestration, from 0s, x0.25) early-grasp(orchestration, from 0s, x0.5) occlude-target(orchestration, from 0s, x1) phantom-actor(orchestration, from 0s, x0.5) phantom-actor(orchestration, from 0s, x1) remove-actor(orchestration, from 0s, x0.5) remove-actor(orchestration, from 0s, x1) stall(orchestration, from 0s, x0.5) stall(orchestration, from 0s, x1) weak-grip(orchestration, from 0s, x0.5) weak-grip(orchestration, from 0s, x0.75) 2. retrieve_the_medicine_kept_as_found +3 defect(s) reorder(orchestration, from 0s, x0.25) reorder(orchestration, from 0s, x0.5) reorder(orchestration, from 0s, x1) ==================================================================== 2 SCENARIOS ADD NOTHING THE SET ABOVE DOES NOT HAVE ==================================================================== Not bad scenarios - duplicates for this catalogue and this system. Widen the catalogue and some of these stop being duplicates, which is why this is a budget observation and not a finding. retrieve_the_medicine 697 run(s) tidy_the_kitchen 470 run(s) 1167 run(s) a sweep, for evidence the set above already gives. ==================================================================== RETIRE THESE AND THE COVERAGE GOES WITH THEM ==================================================================== Exactly one scenario in the suite catches each of these. Nothing else would notice if it went, which is the case every stale-corpus cleanup gets wrong. tidy_the_kitchen_kept_as_found only source of stall(orchestration, from 0s, x0.5) retrieve_the_medicine_kept_as_found only source of reorder(orchestration, from 0s, x0.25) only source of reorder(orchestration, from 0s, x0.5) only source of reorder(orchestration, from 0s, x1) ==================================================================== 4 SCENARIOS HAVE NO HEADROOM LEFT ==================================================================== Each of these still catches the defect named, and catches it only at the severest magnitude the catalogue contains. One improvement of that size and the coverage goes. None of it has gone yet - this is where the next loss will be, and it is the only warning there is. retrieve_the_medicine depth-error only at x0.5 of 0.25 / 0.5 retrieve_the_medicine_kept_as_found stall only at x1 of 0.25 / 0.5 / 1 depth-error only at x0.5 of 0.25 / 0.5 tidy_the_kitchen occlude-target only at x1 of 0.25 / 0.5 / 1 depth-error only at x0.5 of 0.25 / 0.5 tidy_the_kitchen_kept_as_found occlude-target only at x1 of 0.25 / 0.5 / 1 depth-error only at x0.5 of 0.25 / 0.5 ==================================================================== CHANGE SINCE 2026-09-30 22:42 (published+kept-as-found) ==================================================================== discrimination 55.2% -> 49.5% (-5.7 points) ==================================================================== 4 SCENARIOS HAVE STOPPED TESTING WHAT THEY DID ==================================================================== These still run and still pass. What they no longer do is notice the defect they were kept for. retrieve_the_medicine 50.0% -> 43.5% (-6.5) no longer catches unsteady(orchestration, from 0s, x0.3) still catches phantom-actor, but now needs x0.5 where x0.25 used to be enough retrieve_the_medicine_kept_as_found 64.0% -> 58.3% (-5.7) no longer catches unsteady(orchestration, from 0s, x0.3) still catches phantom-actor, but now needs x0.5 where x0.25 used to be enough tidy_the_kitchen_kept_as_found 57.1% -> 51.7% (-5.4) no longer catches phantom-actor(orchestration, from 0s, x0.25) still catches phantom-actor, but now needs x0.5 where x0.25 used to be enough tidy_the_kitchen 50.0% -> 44.8% (-5.2) no longer catches phantom-actor(orchestration, from 0s, x0.25) still catches phantom-actor, but now needs x0.5 where x0.25 used to be enough ==================================================================== THE SUITE NO LONGER COVERS THESE AT ALL ==================================================================== Nothing left in the corpus catches these. Every other finding above is a scenario worth retiring; this one is a hole, and retiring anything will not close it. phantom-actor(orchestration, from 0s, x0.25) unsteady(orchestration, from 0s, x0.3) mutational stale: the suite catches less than it did. This is usually a release that improved the system, and the scenarios above are no longer earning the compute they cost. stale --gate exited 1