Sample pilot report

In one paragraph

The two published demonstrations, tidy the kitchen and retrieve the medicine, pass every clean run. Under 48 decided faults they caught 26 - and all 26 failed through one check, whether the task finished. Ten faults were noticed by nothing, including a model twenty seconds slower on every decision. Two scenes, the same scenes asserting four more things, earned their place and closed twelve items on the worklist. Then a better build - one that checks its own hand after a pick - failed the gate: nothing in the suite had changed, every scenario still passed, and all four had stopped testing what they tested before. And a rehearsal in a twin before moving said go on 12 of the 16 faults the real run failed on in the kitchen.

26 / 26

catches that failed through one check, task_incomplete

10

faults no scenario noticed, of the ones that reached a verdict

12 / 16

real failures the rehearsal said go on, tidying the kitchen

1. What was measured

2. What the suite catches

Every catch failed through one check. All 26 catches that said what failed, failed task_incomplete. That is what a single assertion looks like from the outside: whatever a fault does on the way that the task did not name, nothing notices. (1-stale.txt)

Twelve faults were noticed by nothing: short sight (blind) at all three magnitudes, a slow model (stall) at all three, weak actuation (sluggish) at both, a stereo depth bias at the mildest, an unsteady release at the mildest, and the plan carried out in the wrong order (reorder) at the two larger magnitudes - the carton to the bin before the medicine to the person, with both still finished. Each was injected, reached a verdict, and changed no outcome anywhere.

One fault could not be delivered at all. A late memory refresh (lag) was refused in every scenario, correctly: nothing in either kitchen moves after exploration, so a stale memory is the same memory. The suite has no scene in which that fault could matter (1-reach.txt).

3. What to add

The worklist (2-next.txt) named four hazards whose scenes exist and whose assertions do not: a false completion report, left holding, object dropped, and too slow. For bystander contact it asked for a scene with a person in it rather than a check in a kitchen with nobody in it. And it named the right actions in the wrong order: a fault reaches it, and no scenario asserts anything about order.

The same two scenes, asserting those checks - and the medicine scene stating that the medicine goes to the person before the carton goes to the bin, our reading of the instruction rather than the page's - were judged by mutational accept (3-accept.txt): both earn their place. Tidy is the only evidence against leaving the kitchen unsafe and the only scenario that catches a model ten seconds slower per decision; medicine is the only evidence against a dropped object and against steps done out of order. Fifteen worklist items closed, and eleven new ones opened.

Over 40 seeds each (5-checks-standin.txt).

4. The gate, on a better build

homebody-standin-rechecks sees its empty hand and retries, and sees a misplaced release and fixes it. Swept against the history with the gate on, it exits 1: all four scenarios stopped catching a fault they caught before - the false "holding" and the unsteady release - and in the gate's words, the suite no longer covers these at all (4-stale-gate.txt).

Nothing in the suite changed, and every scenario still runs and passes. A pass-rate dashboard shows a better release. What the gate shows is that the scenarios which used to exercise that seam no longer do, so the next regression there would ship unnoticed. That is the decision a pilot gives a team: accept the loss knowingly, or write the scene that still exercises it.

5. Checks that stand in for the real one

A rehearsal in a twin, before moving. Swept as a scenario, with the robot's faults delivered to the robot and the real run still happening, the rehearsal is scored as missing whatever it said go on (6-standin.txt, 6-stale-standins.txt):

It stopped what the twin contains - a forgotten keyframe, a slow model, a plan taken out of order - and said go on everything of the body and the sensors: false grasps, drift, weak grip, early close. Where a bag moved after exploration, short sight at every magnitude and a late memory refresh at the two larger ones failed every real run, and the rehearsal, planning against where the bag used to be, said go every time. Which faults a twin contains is an assumption this adapter states once; for a real twin, it is what the sweep finds out.

A judge of success. Scored on what the robot reported, beside what happened: in tidy the real run caught 14 faults and the report said so on 9; 5 times it claimed success and the task had failed. In medicine, 12, 6 and 6. A robot does not report what it does not know it got wrong.

6. What we would do next

• Assert more than task completion. The two additions above, or your own versions of them. One clause is one question, and the worklist names the four others this corpus never asks.
• Add a scene where something moved. Without one, a stale memory cannot be tested at all, and the rehearsal looks better than it is.
• Put the gate in CI against the reference build, so a release that stops a scenario testing something is a decision somebody makes rather than one that happens.
• Do not ship on the rehearsal alone for faults of the body, until a sweep of your twin says it contains them.

7. What this cannot say

• Nothing about HomeBody. The planner is not GPT Astra, and a model may notice its empty hand the way the rechecking system does. Which of the two HomeBody resembles is what a sweep of the real one would settle.
• Nothing about grasping physics. The tolerances decide which faults land, and they are guesses.
• Nothing reachable only through language - a misread goal. The catalogue declares it unreachable, and no figure above is about it.
• About order, one reading of one instruction. Steps out of order are evidenced only where a scene states an order, and the one that does is our reading of the medicine request, not HomeBody's.
• Nothing about whether catching a fault in simulation predicts catching it on the robot. That is a separate and harder question, and the step after this is a staged rollout with a person.

Your own

A pilot is this, on your simulator, your scenarios and your fault model, run where your simulator is: nothing leaves your machines but names, verdicts and counts. Book a pilot, or start on your own.