mutational: 1 gated hazard(s) are unreachable by "homebody orchestration"; nothing below says anything about them ==================================================================== MUTATIONAL REPORT ==================================================================== runner exec: python3 -m mutational_homebody.runner {scenario} (recording) simulation runs 5818 in 26.136s settled early 20 inconclusive verdicts reached before their last trial, which could not have changed them, and 99 runs not spent. --curtail=false runs them discrimination 38.8% (109 of 281 injected defects detected) over the 281 defects decided, not the 410 planned: 53 defects have no verdict yet, so this samples the discrimination rather than measuring it scenarios 10 scored, 0 skipped, 29 inconclusive defects undeliverable 76 defects could not be expressed in the scenarios aimed at, and are excluded: not exposed, and not missed either. rarely delivered 24 defects landed on some placements and were refused on most, and ran out of attempts before there were trials enough to judge: tidy_after_a_bag_was_moved_rehearsed reorder(orchestration, from 0s, x0.25) landed on 12 of 40 tidy_the_kitchen_rehearsed reorder(orchestration, from 0s, x0.25) landed on 12 of 40 retrieve_the_medicine_rehearsed occlude-target(orchestration, from 0s, x0.25) landed on 19 of 40 retrieve_the_medicine_rehearsed myopic(orchestration, from 0s, x0.25) landed on 6 of 40 retrieve_the_medicine_rehearsed myopic(orchestration, from 0s, x0.5) landed on 6 of 40 ... and 19 more No verdict, and not refused. More --trials buys more placements. ==================================================================== PER SCENARIO ==================================================================== tidy_after_a_bag_was_moved_rehearsed 12.5% 4/32 flake 0% (32 of 41 planned) tidy_the_kitchen_rehearsed 13.8% 4/29 flake 0% (29 of 41 planned) retrieve_the_medicine_rehearsed 20.7% 6/29 flake 0% (29 of 41 planned) retrieve_the_medicine_judged 24.0% 6/25 flake 0% (25 of 41 planned) tidy_the_kitchen_judged 30.0% 9/30 flake 0% (30 of 41 planned) retrieve_the_medicine 50.0% 12/24 flake 0% (24 of 41 planned) tidy_the_kitchen 50.0% 14/28 flake 0% (28 of 41 planned) tidy_the_kitchen_kept_as_found 57.1% 16/28 flake 0% (28 of 41 planned) retrieve_the_medicine_kept_as_found 64.0% 16/25 flake 0% (25 of 41 planned) tidy_after_a_bag_was_moved 71.0% 22/31 flake 0% (31 of 41 planned) ==================================================================== UNDETECTED DEFECTS ==================================================================== tidy_after_a_bag_was_moved_rehearsed phantom-actor(orchestration, from 0s, x0.25) tidy_after_a_bag_was_moved_rehearsed phantom-actor(orchestration, from 0s, x0.5) tidy_after_a_bag_was_moved_rehearsed phantom-actor(orchestration, from 0s, x1) tidy_after_a_bag_was_moved_rehearsed occlude-target(orchestration, from 0s, x0.25) ... 24 more in this scenario tidy_the_kitchen_rehearsed phantom-actor(orchestration, from 0s, x0.25) tidy_the_kitchen_rehearsed phantom-actor(orchestration, from 0s, x0.5) tidy_the_kitchen_rehearsed phantom-actor(orchestration, from 0s, x1) tidy_the_kitchen_rehearsed occlude-target(orchestration, from 0s, x0.25) ... 21 more in this scenario retrieve_the_medicine_rehearsed phantom-actor(orchestration, from 0s, x0.25) retrieve_the_medicine_rehearsed phantom-actor(orchestration, from 0s, x0.5) retrieve_the_medicine_rehearsed phantom-actor(orchestration, from 0s, x1) retrieve_the_medicine_rehearsed occlude-target(orchestration, from 0s, x0.5) ... 19 more in this scenario retrieve_the_medicine_judged phantom-actor(orchestration, from 0s, x0.25) retrieve_the_medicine_judged phantom-actor(orchestration, from 0s, x0.5) retrieve_the_medicine_judged phantom-actor(orchestration, from 0s, x1) retrieve_the_medicine_judged remove-actor(orchestration, from 0s, x0.25) ... 15 more in this scenario tidy_the_kitchen_judged phantom-actor(orchestration, from 0s, x0.25) tidy_the_kitchen_judged phantom-actor(orchestration, from 0s, x0.5) tidy_the_kitchen_judged phantom-actor(orchestration, from 0s, x1) tidy_the_kitchen_judged occlude-target(orchestration, from 0s, x0.25) ... 17 more in this scenario retrieve_the_medicine depth-error(orchestration, from 0s, x0.25) retrieve_the_medicine blind(orchestration, from 0s, x0.25) retrieve_the_medicine blind(orchestration, from 0s, x0.5) retrieve_the_medicine blind(orchestration, from 0s, x0.75) ... 8 more in this scenario tidy_the_kitchen occlude-target(orchestration, from 0s, x0.25) tidy_the_kitchen occlude-target(orchestration, from 0s, x0.5) tidy_the_kitchen depth-error(orchestration, from 0s, x0.25) tidy_the_kitchen blind(orchestration, from 0s, x0.25) ... 10 more in this scenario tidy_the_kitchen_kept_as_found occlude-target(orchestration, from 0s, x0.25) tidy_the_kitchen_kept_as_found occlude-target(orchestration, from 0s, x0.5) tidy_the_kitchen_kept_as_found depth-error(orchestration, from 0s, x0.25) tidy_the_kitchen_kept_as_found blind(orchestration, from 0s, x0.25) ... 8 more in this scenario retrieve_the_medicine_kept_as_found depth-error(orchestration, from 0s, x0.25) retrieve_the_medicine_kept_as_found blind(orchestration, from 0s, x0.25) retrieve_the_medicine_kept_as_found blind(orchestration, from 0s, x0.5) retrieve_the_medicine_kept_as_found blind(orchestration, from 0s, x0.75) ... 5 more in this scenario tidy_after_a_bag_was_moved occlude-target(orchestration, from 0s, x0.25) tidy_after_a_bag_was_moved occlude-target(orchestration, from 0s, x0.5) tidy_after_a_bag_was_moved lag(orchestration, from 0s, x0.1) tidy_after_a_bag_was_moved depth-error(orchestration, from 0s, x0.25) ... 5 more in this scenario ==================================================================== WHAT THIS SUITE IS PAYING FOR ==================================================================== scenarios 10 scored defects caught 26 noticed by none 8 more, injected into several and caught in none scenarios needed 2 of 10 to catch all of them simulator runs 5818 this sweep ==================================================================== 8 DEFECTS WERE NOTICED BY NOTHING IN THIS SUITE ==================================================================== Each was injected into several scenarios, reached a verdict in all of them, and changed no outcome anywhere. Two things look exactly like this and this run cannot tell them apart: the suite shares a blind spot - a hole worth filling it cannot be caught here at all - worth taking out of the catalogue, not out of the suite Either way it sits in every scenario's denominator and none of them can score on it, so it lowers every figure above. depth-error(orchestration, from 0s, x0.25) 10 scenario(s), 10 of which catch other defects sluggish(orchestration, from 0s, x0.25) 10 scenario(s), 10 of which catch other defects sluggish(orchestration, from 0s, x0.5) 10 scenario(s), 10 of which catch other defects unsteady(orchestration, from 0s, x0.1) 10 scenario(s), 10 of which catch other defects occlude-target(orchestration, from 0s, x0.5) 7 scenario(s), 7 of which catch other defects occlude-target(orchestration, from 0s, x0.25) 6 scenario(s), 6 of which catch other defects lag(orchestration, from 0s, x0.1) 2 scenario(s), 2 of which catch other defects remove-actor(orchestration, from 0s, x0.25) 2 scenario(s), 2 of which catch other defects ==================================================================== THE SET THAT COVERS EVERYTHING THIS SUITE COVERS ==================================================================== Ordered by what each adds that nothing before it did. Stop where the next scenario stops earning its place in the merge queue. 1. tidy_after_a_bag_was_moved +22 defect(s) blind(orchestration, from 0s, x0.25) blind(orchestration, from 0s, x0.5) blind(orchestration, from 0s, x0.75) depth-error(orchestration, from 0s, x0.5) drift(orchestration, from 0s, x0.25) drift(orchestration, from 0s, x0.5) drift(orchestration, from 0s, x0.75) early-grasp(orchestration, from 0s, x0.25) early-grasp(orchestration, from 0s, x0.5) lag(orchestration, from 0s, x0.3) lag(orchestration, from 0s, x0.6) occlude-target(orchestration, from 0s, x1) phantom-actor(orchestration, from 0s, x0.25) phantom-actor(orchestration, from 0s, x0.5) phantom-actor(orchestration, from 0s, x1) remove-actor(orchestration, from 0s, x0.5) remove-actor(orchestration, from 0s, x1) stall(orchestration, from 0s, x0.25) stall(orchestration, from 0s, x0.5) stall(orchestration, from 0s, x1) weak-grip(orchestration, from 0s, x0.5) weak-grip(orchestration, from 0s, x0.75) 2. retrieve_the_medicine_kept_as_found +4 defect(s) reorder(orchestration, from 0s, x0.25) reorder(orchestration, from 0s, x0.5) reorder(orchestration, from 0s, x1) unsteady(orchestration, from 0s, x0.3) ==================================================================== 8 SCENARIOS ADD NOTHING THE SET ABOVE DOES NOT HAVE ==================================================================== Not bad scenarios - duplicates for this catalogue and this system. Widen the catalogue and some of these stop being duplicates, which is why this is a budget observation and not a finding. retrieve_the_medicine 625 run(s) retrieve_the_medicine_judged 781 run(s) retrieve_the_medicine_rehearsed 750 run(s) tidy_after_a_bag_was_moved_rehearsed 713 run(s) tidy_the_kitchen 438 run(s) tidy_the_kitchen_judged 564 run(s) tidy_the_kitchen_kept_as_found 402 run(s) tidy_the_kitchen_rehearsed 634 run(s) 4907 run(s) a sweep, for evidence the set above already gives. ==================================================================== RETIRE THESE AND THE COVERAGE GOES WITH THEM ==================================================================== Exactly one scenario in the suite catches each of these. Nothing else would notice if it went, which is the case every stale-corpus cleanup gets wrong. tidy_after_a_bag_was_moved only source of blind(orchestration, from 0s, x0.25) only source of blind(orchestration, from 0s, x0.5) only source of blind(orchestration, from 0s, x0.75) only source of lag(orchestration, from 0s, x0.3) only source of lag(orchestration, from 0s, x0.6) only source of stall(orchestration, from 0s, x0.25) ==================================================================== 8 SCENARIOS HAVE NO HEADROOM LEFT ==================================================================== Each of these still catches the defect named, and catches it only at the severest magnitude the catalogue contains. One improvement of that size and the coverage goes. None of it has gone yet - this is where the next loss will be, and it is the only warning there is. retrieve_the_medicine depth-error only at x0.5 of 0.25 / 0.5 unsteady only at x0.3 of 0.1 / 0.3 retrieve_the_medicine_judged depth-error only at x0.5 of 0.25 / 0.5 retrieve_the_medicine_kept_as_found stall only at x1 of 0.25 / 0.5 / 1 depth-error only at x0.5 of 0.25 / 0.5 unsteady only at x0.3 of 0.1 / 0.3 retrieve_the_medicine_rehearsed stall only at x1 of 0.25 / 0.5 / 1 tidy_after_a_bag_was_moved occlude-target only at x1 of 0.25 / 0.5 / 1 depth-error only at x0.5 of 0.25 / 0.5 tidy_the_kitchen occlude-target only at x1 of 0.25 / 0.5 / 1 depth-error only at x0.5 of 0.25 / 0.5 tidy_the_kitchen_judged occlude-target only at x1 of 0.25 / 0.5 / 1 depth-error only at x0.5 of 0.25 / 0.5 tidy_the_kitchen_kept_as_found occlude-target only at x1 of 0.25 / 0.5 / 1 depth-error only at x0.5 of 0.25 / 0.5 First sweep recorded. Run this again after a release and it will name the scenarios that stopped catching what they used to.