mutational: 1 gated hazard(s) are unreachable by "homebody orchestration"; nothing below says anything about them ==================================================================== MUTATIONAL REPORT ==================================================================== runner exec: python3 -m mutational_homebody.runner {scenario} (recording) simulation runs 1063 in 5.229s settled early 6 inconclusive verdicts reached before their last trial, which could not have changed them, and 31 runs not spent. --curtail=false runs them discrimination 50.0% (26 of 52 injected defects detected) over the 52 defects decided, not the 82 planned: 14 defects have no verdict yet, so this samples the discrimination rather than measuring it scenarios 2 scored, 0 skipped, 8 inconclusive defects undeliverable 16 defects could not be expressed in the scenarios aimed at, and are excluded: not exposed, and not missed either. rarely delivered 6 defects landed on some placements and were refused on most, and ran out of attempts before there were trials enough to judge: retrieve_the_medicine occlude-target(orchestration, from 0s, x0.25) landed on 19 of 40 retrieve_the_medicine myopic(orchestration, from 0s, x0.25) landed on 6 of 40 retrieve_the_medicine myopic(orchestration, from 0s, x0.5) landed on 6 of 40 retrieve_the_medicine myopic(orchestration, from 0s, x0.75) landed on 18 of 40 retrieve_the_medicine reorder(orchestration, from 0s, x0.25) landed on 12 of 40 ... and 1 more No verdict, and not refused. More --trials buys more placements. ==================================================================== PER SCENARIO ==================================================================== retrieve_the_medicine 50.0% 12/24 flake 0% (24 of 41 planned) tidy_the_kitchen 50.0% 14/28 flake 0% (28 of 41 planned) ==================================================================== UNDETECTED DEFECTS ==================================================================== retrieve_the_medicine depth-error(orchestration, from 0s, x0.25) retrieve_the_medicine blind(orchestration, from 0s, x0.25) retrieve_the_medicine blind(orchestration, from 0s, x0.5) retrieve_the_medicine blind(orchestration, from 0s, x0.75) ... 8 more in this scenario tidy_the_kitchen occlude-target(orchestration, from 0s, x0.25) tidy_the_kitchen occlude-target(orchestration, from 0s, x0.5) tidy_the_kitchen depth-error(orchestration, from 0s, x0.25) tidy_the_kitchen blind(orchestration, from 0s, x0.25) ... 10 more in this scenario ==================================================================== WHAT THIS SUITE IS PAYING FOR ==================================================================== scenarios 2 scored defects caught 15 noticed by none 12 more, injected into several and caught in none scenarios needed 2 of 2 to catch all of them simulator runs 1063 this sweep ==================================================================== 12 DEFECTS WERE NOTICED BY NOTHING IN THIS SUITE ==================================================================== Each was injected into several scenarios, reached a verdict in all of them, and changed no outcome anywhere. Two things look exactly like this and this run cannot tell them apart: the suite shares a blind spot - a hole worth filling it cannot be caught here at all - worth taking out of the catalogue, not out of the suite Either way it sits in every scenario's denominator and none of them can score on it, so it lowers every figure above. blind(orchestration, from 0s, x0.25) 2 scenario(s), 2 of which catch other defects blind(orchestration, from 0s, x0.5) 2 scenario(s), 2 of which catch other defects blind(orchestration, from 0s, x0.75) 2 scenario(s), 2 of which catch other defects depth-error(orchestration, from 0s, x0.25) 2 scenario(s), 2 of which catch other defects reorder(orchestration, from 0s, x0.5) 2 scenario(s), 2 of which catch other defects reorder(orchestration, from 0s, x1) 2 scenario(s), 2 of which catch other defects sluggish(orchestration, from 0s, x0.25) 2 scenario(s), 2 of which catch other defects sluggish(orchestration, from 0s, x0.5) 2 scenario(s), 2 of which catch other defects stall(orchestration, from 0s, x0.25) 2 scenario(s), 2 of which catch other defects stall(orchestration, from 0s, x0.5) 2 scenario(s), 2 of which catch other defects stall(orchestration, from 0s, x1) 2 scenario(s), 2 of which catch other defects unsteady(orchestration, from 0s, x0.1) 2 scenario(s), 2 of which catch other defects ==================================================================== THE SET THAT COVERS EVERYTHING THIS SUITE COVERS ==================================================================== Ordered by what each adds that nothing before it did. Stop where the next scenario stops earning its place in the merge queue. 1. tidy_the_kitchen +14 defect(s) depth-error(orchestration, from 0s, x0.5) drift(orchestration, from 0s, x0.25) drift(orchestration, from 0s, x0.5) drift(orchestration, from 0s, x0.75) early-grasp(orchestration, from 0s, x0.25) early-grasp(orchestration, from 0s, x0.5) occlude-target(orchestration, from 0s, x1) phantom-actor(orchestration, from 0s, x0.25) phantom-actor(orchestration, from 0s, x0.5) phantom-actor(orchestration, from 0s, x1) remove-actor(orchestration, from 0s, x0.5) remove-actor(orchestration, from 0s, x1) weak-grip(orchestration, from 0s, x0.5) weak-grip(orchestration, from 0s, x0.75) 2. retrieve_the_medicine +1 defect(s) unsteady(orchestration, from 0s, x0.3) ==================================================================== RETIRE THESE AND THE COVERAGE GOES WITH THEM ==================================================================== Exactly one scenario in the suite catches each of these. Nothing else would notice if it went, which is the case every stale-corpus cleanup gets wrong. tidy_the_kitchen only source of drift(orchestration, from 0s, x0.25) only source of drift(orchestration, from 0s, x0.5) only source of drift(orchestration, from 0s, x0.75) retrieve_the_medicine only source of unsteady(orchestration, from 0s, x0.3) ==================================================================== 2 SCENARIOS HAVE NO HEADROOM LEFT ==================================================================== Each of these still catches the defect named, and catches it only at the severest magnitude the catalogue contains. One improvement of that size and the coverage goes. None of it has gone yet - this is where the next loss will be, and it is the only warning there is. retrieve_the_medicine depth-error only at x0.5 of 0.25 / 0.5 unsteady only at x0.3 of 0.1 / 0.3 tidy_the_kitchen occlude-target only at x1 of 0.25 / 0.5 / 1 depth-error only at x0.5 of 0.25 / 0.5 ==================================================================== EVERY CATCH IN THIS SWEEP FAILED THROUGH ONE CHECK ==================================================================== All 26 catches that said what failed, failed `task_incomplete`. That is what an oracle with one clause looks like: the scenes differ, the faults differ, and the only thing any of them can fail on is the same. Whatever else a fault does on the way - to anything the task did not name - nothing in this corpus would notice. More scenarios with the same oracle do not change that; another clause does. First sweep recorded. Run this again after a release and it will name the scenarios that stopped catching what they used to.