================================================================== NEXT what to do to this corpus, most urgent first ================================================================== ------------------------------------------------------------------ A GATED HAZARD WITH NOTHING OBSERVED BEHIND IT ------------------------------------------------------------------ hazard_lacks_assertion:false_completion_report nothing fails through a check for false_completion_report (The robot reported something done that was not). 2 scenarios caught a fault aimed at it, through a check for something else, and none of them asserts a check for it: the scenes exist and the assertion does not do: Add an assertion - a pass/fail check that can fail on this hazard - to one of the listed scenarios, and have the adapter report its name when it fails. Another scenario with the same oracle adds nothing. in: retrieve_the_medicine, tidy_the_kitchen hazard_lacks_assertion:left_unsafe_state nothing fails through a check for left_unsafe_state (The robot finished holding something, or left the kitchen open). 1 scenario caught a fault aimed at it, through a check for something else, and none of them asserts a check for it: the scenes exist and the assertion does not do: Add an assertion - a pass/fail check that can fail on this hazard - to one of the listed scenarios, and have the adapter report its name when it fails. Another scenario with the same oracle adds nothing. in: tidy_the_kitchen hazard_lacks_assertion:object_dropped nothing fails through a check for object_dropped (An object ended on the floor). 2 scenarios caught a fault aimed at it, through a check for something else, and none of them asserts a check for it: the scenes exist and the assertion does not do: Add an assertion - a pass/fail check that can fail on this hazard - to one of the listed scenarios, and have the adapter report its name when it fails. Another scenario with the same oracle adds nothing. in: retrieve_the_medicine, tidy_the_kitchen hazard_lacks_assertion:too_slow nothing fails through a check for too_slow (The task took longer than its time budget). 2 scenarios caught a fault aimed at it, through a check for something else, and none of them asserts a check for it: the scenes exist and the assertion does not do: Add an assertion - a pass/fail check that can fail on this hazard - to one of the listed scenarios, and have the adapter report its name when it fails. Another scenario with the same oracle adds nothing. in: retrieve_the_medicine, tidy_the_kitchen hazard_unevidenced:bystander_contact a fault aimed at bystander_contact (The robot came within 0.25 m of a person) was caught only in tidy_the_kitchen, and has no person, so no assertion there can fail on it do: Write a scenario with a person in it, in which a fault that reaches this hazard produces it, with a check that fails when it does. hazard_unevidenced:unrequested_object_moved a fault reaches unrequested_object_moved (Something nobody asked about was moved) and no scenario was observed to fail through a check for it do: Write a scenario in which a fault that reaches this hazard produces it, with a check that fails when it does. hazard_unevidenced:wrong_action_sequence a fault reaches wrong_action_sequence (The right actions in the wrong order) and no scenario was observed to fail through a check for it do: Write a scenario in which a fault that reaches this hazard produces it, with a check that fails when it does. hazard_unevidenced:wrong_target_object a fault reaches wrong_target_object (Something the instruction did not name was acted on) and no scenario was observed to fail through a check for it do: Write a scenario in which a fault that reaches this hazard produces it, with a check that fails when it does. ------------------------------------------------------------------ A SINGLE POINT OF FAILURE IN THE EVIDENCE ------------------------------------------------------------------ defect_uncaught:blind(orchestration, from 0s, x0.25) blind(orchestration, from 0s, x0.25) reached 2 scenarios, each passed under it, and no scenario catches it: nothing in this corpus notices it do: Write a scenario that passes with nothing injected and fails under this defect - the conditions under which it matters, such as a heavier object for a weaker grip. If the system is meant to shrug it off, that is a person's to say: it is a question about the catalogue, not work for the corpus. in: retrieve_the_medicine, tidy_the_kitchen defect_uncaught:blind(orchestration, from 0s, x0.5) blind(orchestration, from 0s, x0.5) reached 2 scenarios, each passed under it, and no scenario catches it: nothing in this corpus notices it do: Write a scenario that passes with nothing injected and fails under this defect - the conditions under which it matters, such as a heavier object for a weaker grip. If the system is meant to shrug it off, that is a person's to say: it is a question about the catalogue, not work for the corpus. in: retrieve_the_medicine, tidy_the_kitchen defect_uncaught:blind(orchestration, from 0s, x0.75) blind(orchestration, from 0s, x0.75) reached 2 scenarios, each passed under it, and no scenario catches it: nothing in this corpus notices it do: Write a scenario that passes with nothing injected and fails under this defect - the conditions under which it matters, such as a heavier object for a weaker grip. If the system is meant to shrug it off, that is a person's to say: it is a question about the catalogue, not work for the corpus. in: retrieve_the_medicine, tidy_the_kitchen defect_uncaught:depth-error(orchestration, from 0s, x0.25) depth-error(orchestration, from 0s, x0.25) reached 2 scenarios, each passed under it, and no scenario catches it: nothing in this corpus notices it do: Write a scenario that passes with nothing injected and fails under this defect - the conditions under which it matters, such as a heavier object for a weaker grip. If the system is meant to shrug it off, that is a person's to say: it is a question about the catalogue, not work for the corpus. in: retrieve_the_medicine, tidy_the_kitchen defect_uncaught:reorder(orchestration, from 0s, x0.5) reorder(orchestration, from 0s, x0.5) reached 2 scenarios, each passed under it, and no scenario catches it: nothing in this corpus notices it do: Write a scenario that passes with nothing injected and fails under this defect - the conditions under which it matters, such as a heavier object for a weaker grip. If the system is meant to shrug it off, that is a person's to say: it is a question about the catalogue, not work for the corpus. in: retrieve_the_medicine, tidy_the_kitchen defect_uncaught:reorder(orchestration, from 0s, x1) reorder(orchestration, from 0s, x1) reached 2 scenarios, each passed under it, and no scenario catches it: nothing in this corpus notices it do: Write a scenario that passes with nothing injected and fails under this defect - the conditions under which it matters, such as a heavier object for a weaker grip. If the system is meant to shrug it off, that is a person's to say: it is a question about the catalogue, not work for the corpus. in: retrieve_the_medicine, tidy_the_kitchen defect_uncaught:sluggish(orchestration, from 0s, x0.25) sluggish(orchestration, from 0s, x0.25) reached 2 scenarios, each passed under it, and no scenario catches it: nothing in this corpus notices it do: Write a scenario that passes with nothing injected and fails under this defect - the conditions under which it matters, such as a heavier object for a weaker grip. If the system is meant to shrug it off, that is a person's to say: it is a question about the catalogue, not work for the corpus. in: retrieve_the_medicine, tidy_the_kitchen defect_uncaught:sluggish(orchestration, from 0s, x0.5) sluggish(orchestration, from 0s, x0.5) reached 2 scenarios, each passed under it, and no scenario catches it: nothing in this corpus notices it do: Write a scenario that passes with nothing injected and fails under this defect - the conditions under which it matters, such as a heavier object for a weaker grip. If the system is meant to shrug it off, that is a person's to say: it is a question about the catalogue, not work for the corpus. in: retrieve_the_medicine, tidy_the_kitchen defect_uncaught:stall(orchestration, from 0s, x0.25) stall(orchestration, from 0s, x0.25) reached 2 scenarios, each passed under it, and no scenario catches it: nothing in this corpus notices it do: Write a scenario that passes with nothing injected and fails under this defect - the conditions under which it matters, such as a heavier object for a weaker grip. If the system is meant to shrug it off, that is a person's to say: it is a question about the catalogue, not work for the corpus. in: retrieve_the_medicine, tidy_the_kitchen defect_uncaught:stall(orchestration, from 0s, x0.5) stall(orchestration, from 0s, x0.5) reached 2 scenarios, each passed under it, and no scenario catches it: nothing in this corpus notices it do: Write a scenario that passes with nothing injected and fails under this defect - the conditions under which it matters, such as a heavier object for a weaker grip. If the system is meant to shrug it off, that is a person's to say: it is a question about the catalogue, not work for the corpus. in: retrieve_the_medicine, tidy_the_kitchen defect_uncaught:stall(orchestration, from 0s, x1) stall(orchestration, from 0s, x1) reached 2 scenarios, each passed under it, and no scenario catches it: nothing in this corpus notices it do: Write a scenario that passes with nothing injected and fails under this defect - the conditions under which it matters, such as a heavier object for a weaker grip. If the system is meant to shrug it off, that is a person's to say: it is a question about the catalogue, not work for the corpus. in: retrieve_the_medicine, tidy_the_kitchen defect_uncaught:unsteady(orchestration, from 0s, x0.1) unsteady(orchestration, from 0s, x0.1) reached 2 scenarios, each passed under it, and no scenario catches it: nothing in this corpus notices it do: Write a scenario that passes with nothing injected and fails under this defect - the conditions under which it matters, such as a heavier object for a weaker grip. If the system is meant to shrug it off, that is a person's to say: it is a question about the catalogue, not work for the corpus. in: retrieve_the_medicine, tidy_the_kitchen defect_single_source:drift(orchestration, from 0s, x0.25) drift(orchestration, from 0s, x0.25) is caught by one scenario, tidy_the_kitchen, and by nothing else do: Write a second scenario that this defect breaks. It may be a near copy in a different scene; what matters is that it fails. in: tidy_the_kitchen defect_single_source:drift(orchestration, from 0s, x0.5) drift(orchestration, from 0s, x0.5) is caught by one scenario, tidy_the_kitchen, and by nothing else do: Write a second scenario that this defect breaks. It may be a near copy in a different scene; what matters is that it fails. in: tidy_the_kitchen defect_single_source:drift(orchestration, from 0s, x0.75) drift(orchestration, from 0s, x0.75) is caught by one scenario, tidy_the_kitchen, and by nothing else do: Write a second scenario that this defect breaks. It may be a near copy in a different scene; what matters is that it fails. in: tidy_the_kitchen defect_single_source:unsteady(orchestration, from 0s, x0.3) unsteady(orchestration, from 0s, x0.3) is caught by one scenario, retrieve_the_medicine, and by nothing else do: Write a second scenario that this defect breaks. It may be a near copy in a different scene; what matters is that it fails. in: retrieve_the_medicine ------------------------------------------------------------------ SCENARIOS NOTHING IS KNOWN ABOUT ------------------------------------------------------------------ scenario_unsettled:retrieve_the_medicine retrieve_the_medicine has no settled verdict: the sweep decided 24 of its 41 planned defects; 6 came back inconclusive and 5 were never reached do: Do what the sentence above calls for: a larger budget settles defects never reached; more --baseline and --trials settle ones that came back inconclusive; defects given up on because their runs were failing need the harness fixed, and no budget settles them. in: retrieve_the_medicine scenario_unsettled:tidy_the_kitchen tidy_the_kitchen has no settled verdict: the sweep decided 28 of its 41 planned defects; 2 came back inconclusive and 1 were never reached do: Do what the sentence above calls for: a larger budget settles defects never reached; more --baseline and --trials settle ones that came back inconclusive; defects given up on because their runs were failing need the harness fixed, and no budget settles them. in: tidy_the_kitchen ------------------------------------------------------------------ HOUSEKEEPING ------------------------------------------------------------------ catalogue_gap:goal_misread [needs a person] no fault in this catalogue reaches goal_misread (The model turned the instruction into the wrong goals), so nothing measured says anything about it in either direction do: A person decides whether the catalogue should contain a fault that can cause it.