From install to a gate in CI
Start here
Make a free account, make an API token under Tokens, and install the CLI with it: one binary for macOS or Linux, checked against its published sha256 before it is made executable. The account is for downloads and the hosted history; nothing below needs it after that.
export MUTATIONAL_TOKEN=mutational_... curl -fsSL https://mutational.in/install.sh | sh
mutational init writes a project that runs as it is: a standard-library Python adapter around a toy one-dimensional car, three scenarios, a four-hazard catalogue and a project file. Sweep it before you touch it. The canary is caught everywhere, blind is caught nowhere, and next lists what the corpus does not prove yet. Two kinds of answer come back: gaps a scenario or an assertion closes, and gaps no scenario can - the toy car stops within whatever sight blind leaves it, which is a question about the catalogue's magnitudes, not the corpus. Then replace the toy simulate() with a call into your simulator.
The project is named after its directory, and the collector files its sweeps and its push under that name. Replace the toy in place and it stays one corpus: a push under the same name replaces it, where a project renamed beside it is a second corpus - and Open has one. A corpus you no longer want comes off its page on the dashboard, or with mutational project delete --name NAME --history.
mutational init --dir my-sim && cd my-sim mutational doctor --project mutational.project.json # is the adapter measuring anything? mutational stale --project mutational.project.json # what the corpus catches, fault by fault mutational next --project mutational.project.json # what to add, most urgent first # keep the history on the collector, under the project's name mutational stale --project mutational.project.json --post https://collector.mutational.in
Your first sweep
A sweep runs where your simulator is. It makes your system worse on purpose - a fault from a catalogue, through your adapter - and records which scenarios' own checks object. Start with the pre-flight: it finds the adapter mistakes that would otherwise be scored as results.
# is the adapter ready to measure anything, and can this budget detect a fault? mutational doctor --suite suite.json --baseline 3 --trials 5 -- ./run.sh '{scenario}' # sweep, keep every run, post the result to this workspace export MUTATIONAL_TOKEN=mutational_... mutational stale --suite suite.json --baseline 3 --trials 5 \ --record history.jsonl --record-runs runs.jsonl \ --fingerprint "$(git rev-parse --short HEAD)" \ --post https://collector.mutational.in -- ./run.sh '{scenario}'
It appears under Sweeps. What is posted is a summary - scenario names, scores, which faults each caught - and the recording with everything your adapter printed stripped out.
The adapter contract
One script that runs one scenario. Mutational knows nothing else about your stack.
| What | Means |
|---|---|
$1, MUTATIONAL_SCENARIO | the scenario to run |
MUTATIONAL_MUTANT | the fault to inject, as one flat JSON object - or the literal null for a clean run |
MUTATIONAL_SEED | a seed, so a run is reproducible |
| exit 0 | the scenario passed |
| exit 1 | the scenario failed - it caught this fault |
| exit 3 | the fault means nothing in this scenario; it leaves the denominator |
| anything else | the harness broke. Never counted as a catch |
MUTATIONAL_REASON: ... | optional, one line: on exit 1 the name of the check that failed |
The mistake that matters most: an uncaught exception in a Python adapter exits 1, which reads as a catch. Wrap your entry point and exit 2.
The fault arrives as {"kind": "lag", "magnitude": 0.3, "start_ms": 0, "for_ms": 0}. The kinds: blind, lag, myopic, coarse, unsteady, sluggish, stall on the policy; occlude, dropout, degrade, remove-actor, appearance, occlude-target, phantom-actor on perception; weak-grip, depth-error, early-grasp, drift, yaw-error on manipulation; camera-shift, friction, mass, compliance, backlash on the robot's physics and its camera (compliance at 1 is a joint with no stiffness: a power or actuator loss). drop is the canary: the whole observation removed. regression carries a build instead of a magnitude: a build that shipped and was confirmed bad, to run clean in place of the one under test - refuse it unless your adapter can load an archived build, and exit 2 for one it cannot find. Deliver the ones your stack can express, and refuse the rest by name with exit 3.
A minimal adapter
The shape of every adapter, with your simulator where run_in_my_simulator is. mutational init writes a longer one, which also checks the fault's fields and prints the suite and facts files.
#!/usr/bin/env python3
import json, os, sys, traceback
DELIVERED = {"blind", "lag", "sluggish"} # what your stack can express
CANARY = "drop" # the whole observation removed
def said(code, why):
print("MUTATIONAL_REASON: " + " ".join(str(why).split()), file=sys.stderr)
return code
def main():
name = sys.argv[1] if len(sys.argv) > 1 else os.environ["MUTATIONAL_SCENARIO"]
fault = json.loads(os.environ.get("MUTATIONAL_MUTANT") or "null")
seed = int(os.environ.get("MUTATIONAL_SEED", "1"))
if fault and fault["kind"] not in DELIVERED | {CANARY}:
return said(3, fault["kind"] + " is not wired into this adapter")
# yours: run one scenario with the fault in the loop, and return the checks
# that failed (empty if it passed) and whether the fault changed anything
failed, landed = run_in_my_simulator(name, fault, seed)
if fault and not landed:
return said(3, "the fault changed nothing in this scenario")
if not failed:
return 0
if len(failed) > 1:
print("MUTATIONAL_CHECKS: " + " ".join(failed), file=sys.stderr)
return said(1, failed[0]) # the first check that failed
if __name__ == "__main__":
try:
sys.exit(main())
except SystemExit:
raise
except BaseException: # a crash is never a catch
traceback.print_exc()
sys.exit(said(2, "the adapter broke"))The suite file
What to sweep. One entry per scenario: its name, which your adapter is given, and where a fault may be delivered. That is all mutational needs; the scenario itself stays in your simulator's own format.
{ "scenarios": [
{ "name": "unprotected_left",
"targets": [{ "channel": "policy" }],
"source": "scenarios/unprotected_left.xosc" },
{ "name": "too_wide_to_grasp",
"targets": [{ "channel": "policy" }],
"expect": "fail", "expect_checks": ["grasp_failed"] }
] }| Field | Means |
|---|---|
name | required; unique in the suite |
targets | where faults go: a channel (policy, a camera, a stream) and optionally a field of it |
source | the file that defines it. Hashed, so editing one scenario re-scores that scenario and nothing else. Never sent anywhere |
fingerprint | in place of source, when you can say better what a verdict depends on - a content hash that also covers the map |
expect, expect_checks | "fail" makes it a positive control: written so the sound system fails it, through these checks. Run clean, never scored; if it passes, your checks cannot see what it was written to show |
stands_in_for, as | a rehearsal or a judge of another scenario: see below |
The facts file
What each scenario asserts and puts in the world, one JSON object a line, written by your adapter (describe.facts in the project file). next, triage, accept and the hosted worklist read it; doctor --project checks that it reads before a campaign is spent.
| Field | Means |
|---|---|
name, enabled | required: the scenario, and whether it runs. Left out, enabled would read as disabled, so it is refused |
actors | what the scene puts in the world, in the words a catalogue's needs uses |
checks, has_oracle | the checks it asserts, and whether it asserts any at all |
overrides | a check this scenario downgrades, to what it downgrades it to, as text: {"collision": "warning"}. Not scenario parameters - those belong in the scenario's own file |
tags | free text, for grouping: a task, a placement, a site |
The catalogue
The fault model, and what your release is gated on. Hazards are the outcomes that fail a release; operators are the faults and what each is aimed at; checks map what your adapter prints as MUTATIONAL_REASON to the hazards a failure through it is evidence against. Without a catalogue, stale uses a built-in set of operators and says nothing about hazards.
{ "name": "urban-av", "version": "3",
"source": "Our safety case, section 4; incident reports 2026-Q2",
"hazards": [
{ "id": "collision", "name": "The vehicle hit something",
"gate": "fail", "needs": ["obstacle"] },
{ "id": "route_complete", "name": "It got where it was going",
"gate": "pass" } ],
"operators": [
{ "name": "lag", "kind": "lag", "magnitudes": [0.1, 0.3, 0.6],
"reaches": ["collision"],
"rationale": "The policy acts on the picture from m seconds ago." } ],
"checks": { "collision": ["collision"] } }| Field | Means |
|---|---|
hazards[].gate | fail: a hazard a release is gated on. pass: its occurrence is success. ignore: observed and deliberately not gated, with a reason |
hazards[].needs | what a scene must have for the hazard to happen there, in the words your facts use for actors. The worklist does not ask for a bystander check in a scene with nobody in it |
operators[].magnitudes | the ladder, mildest first. How far up it a scenario still catches the fault is its headroom |
operators[].windows | optionally, when a fault acts: {"start_ms", "for_ms"}. Omitted, it acts for the whole run |
operators[].reaches | the hazards the fault is aimed at. A catch through a check for something else is not evidence about these |
checks | your adapter's check names, each to the hazards a failure through it evidences. Without this, a catch is held only against what its fault was aimed at, and the report marks it unconfirmed |
Say where each hazard comes from - a standard, an incident, a safety case. A catalogue whose hazards and faults share one author measures that author's imagination.
A gate in CI
Post every build's sweep, then ask whether this build lost coverage. The gate exits 1 when a scenario stopped catching something it used to, and 2 when nobody can say - an unanswered question is never a pass.
# .github/workflows/sim.yml, after your sweep step - run: | mutational gate --url https://collector.mutational.in \ --suite-name suite.json --fingerprint "$(git rev-parse --short HEAD)" env: MUTATIONAL_TOKEN: ${{ secrets.MUTATIONAL_TOKEN }}
--suite-name is the name the sweep was filed under: the suite file's name for stale --suite, the project's name for stale --project.
Hold every build to one you chose
Each build is compared with the sweep posted before it in its series - whatever posted it. With collector in a project file every sweep made from it is posted, a local experiment as much as a release, and the next CI build is compared with that. Pin the build releases are held to, once, with a full token or from the sweep's page on the dashboard: every later build is then also compared with it, and fails the gate on anything it lost against it. When nothing is pinned the gate says what it compared with.
mutational gate --pin --url https://collector.mutational.in \
--suite-name my-project --fingerprint release-1The gate fails on a defect a scenario stopped catching. One it caught before and left undecided this time - inconclusive, or never reached - is not a loss, and is named under the answer, with how the scenario's clean runs moved: a build that makes a scenario flaky shows up there. Compare sweeps made on one platform: a physics engine's arithmetic differs slightly between macOS and Linux, enough to move a borderline verdict, so sweep where CI sweeps.
Rehearsals and success judges
Some robots rehearse a plan in a digital twin before they move, and go only if it passes. Most decide a task succeeded from a judge: a skill's own result, or a vision-language model looking at the scene. Each is a check that stands in for the real one, and each can say go - or done - when the real run fails.
Write the stand-in as a scenario your adapter runs with the fault delivered to the robot, reporting the stand-in's verdict as its own. The real run still happens, so a fault the twin cannot contain is scored as the stand-in missing it. Name the pair in your suite, sweep both in one sweep, and ask:
# suite.json: each stand-in names the scenario it stands in for {"name": "tidy_rehearsed", "stands_in_for": "tidy", "as": "rehearsal", ...} {"name": "tidy_judged", "stands_in_for": "tidy", "as": "judge", ...} # every fault the real run failed on and the stand-in passed mutational standin --suite suite.json --record history.jsonl
It exits 1 when a stand-in said go, or done, on a fault the real run failed on, and 2 when no pair was measured in one sweep. A fault either side left undecided is never counted as a miss.
Record and replay
--record-runs runs.jsonl keeps every simulator run. Replay it anywhere into the full report, with no simulator - on a laptop, in CI, or in the Playground, where it never leaves the tab.
mutational stale --replay runs.jsonl --suite suite.json --baseline 3 --trials 5
A recording answers for the settings it was made with. Ask for others and the replay says how many runs are missing, and exits 2.
Across machines
Where the simulator runs on other machines - a GPU box, a container per slot - start a worker beside each one. It runs the adapter its own project file pins, so the coordinator sends runs, never commands, and refuses a pool whose workers measure with anything different. Name the workers in the project file and stale, doctor and reach all run there; with --suite, pass --remote instead of a command.
# on each simulator machine export MUTATIONAL_WORKER_TOKEN=a-shared-secret mutational worker --project mutational.project.json --listen 0.0.0.0:8791 # mutational.project.json, where the sweep is started "remote": ["http://gpu-1:8791", "http://gpu-2:8791"] mutational doctor --project mutational.project.json mutational stale --project mutational.project.json
Over workers, doctor does not ask an unknown scenario name whether harness failures are visible: a worker refuses a name its suite does not list before any adapter runs. Run doctor with the adapter command on one worker to check that.
Connect an agent (MCP)
mutational mcp serves everything over the Model Context Protocol, on stdio. The rule it enforces is the product: a model may propose anything, and only a sweep accepts it.
# .mcp.json
{ "mcpServers": { "mutational": {
"command": "mutational",
"args": ["mcp", "--project", "mutational.project.json"] } } }Give the agent an agent token from Tokens, not a full one. It can add to your history and read the gate; it cannot delete a sweep or replace one with a sweep that catches less - so it cannot rewrite the history its own work is judged against.
Calibrate the simulator
A sweep says what your scenarios would notice in simulation. Whether a result in simulation stands for one on the real system is another question, and mutational calibrate answers it from outcomes you have on both sides: the same systems and scenarios, run in simulation and on the robot. Each side is a JSONL file, one line per system and scenario. SimCloud writes the simulation side of an evaluation campaign (GET /v1/campaigns/<id>/outcomes?format=mutational); the real side is your lab's or the field's.
# sim.jsonl {"format":"mutational.sim-outcomes/1","system":"pi0@ckpt-40k","scenario":"pick-cube", "successes":17,"trials":20,"source":"simcloud","at":"2026-10-01T12:00:00Z"} # real.jsonl {"format":"mutational.real-outcomes/1","system":"pi0@ckpt-40k","scenario":"pick-cube", "successes":6,"trials":10,"source":"lab","at":"2026-10-02T09:00:00Z"}
It pairs the two and states how far they agree - Pearson's r and Spearman's ρ of the success rates, and the mean maximum rank violation (MMRV), each with its interval, overall and by family - and a verdict read against thresholds it states: trusted, a result in simulation can stand for one on the real system; screen only, it ranks systems as the real one does, enough to screen with and not to decide on; untrusted; or cannot tell, too few pairs or intervals too wide - which is not a pass. A verdict covers the systems and scenarios it paired, and nothing else.
Post it and it is kept beside the corpus's sweeps, under Calibrations, with every interval drawn. It is kept as it was sent; nothing on the collector computes a correlation again.
mutational calibrate --sim sim.jsonl --real real.jsonl --json > calibration.json # filed under the corpus it calibrates curl -fsS -H "Authorization: Bearer $MUTATIONAL_TOKEN" -H "Content-Type: application/json" \ --data-binary @calibration.json \ "https://collector.mutational.in/v1/calibrations?suite=my-sim"
How many trials
A comparison of two systems is only as good as the trials behind it. Before an evaluation, mutational power says how many trials each side needs to tell a difference of the size you care about from noise, at the confidence you ask for; after one, how small a difference the trials you ran could have seen. A comparison run on fewer trials than it needed is one whose "no difference" says nothing, and the document it writes (mutational.power/1) says so before anybody reads the result as one.
Accept, reject or defer
mutational compare decides a new system against the one it would replace, from their successes and trials: it puts a Newcombe interval on the difference between the two success rates and reads it against the margin you set. Accept when the whole interval clears the margin, reject when the whole of it falls short, and defer when it straddles - the answer most often owed, and the one a single success rate never gives. It writes mutational.compare/1, which an evidence pack carries.
Signed evidence
mutational evidence pack puts the evidence a release decision rests on into one document - the calibration it trusts, the comparison it decided by, what the corpus behind it catches, and a regulatory map: which clauses of which frameworks each part of the pack answers - and signs its exact bytes with an Ed25519 key you hold. The signature is a file of its own beside the pack (mutational.evidence-signature/1), and mutational evidence verify checks it on any machine, with the public key and nothing of ours.
Post the pack as its exact text beside its signature: the signature is over those bytes, and a pack written out again would be other bytes.
# the pack's text, as a string, and its signature jq -n --rawfile pack pack.json --slurpfile sig pack.json.sig \ '{pack: $pack, signature: $sig[0]}' \ | curl -fsS -H "Authorization: Bearer $MUTATIONAL_TOKEN" -H "Content-Type: application/json" \ --data-binary @- "https://collector.mutational.in/v1/evidence-packs?suite=my-sim"
Register the public key once, under the id its signatures name - on the Evidence page, or PUT /v1/evidence-keys with a full token - and every pack signed under it is checked as it arrives: verified when its signature checks out, refused when it does not, and kept as not verified when no key is registered under that id. Registering, replacing or removing a key checks its packs again. Only a person registers a key: an agent's token posts packs, and cannot choose which key counts.
curl -fsS -X PUT -H "Authorization: Bearer $MUTATIONAL_TOKEN" -H "Content-Type: application/json" \
-d '{"key_id": "release-ci", "public_key": "<base64url>", "name": "release pipeline"}' \
"https://collector.mutational.in/v1/evidence-keys"Campaigns and the worklist, from the cloud
Push a project and its worklist is here for your team, and for an agent over MCP. Start a campaign from here and it runs on a worker you run next to the simulator. Your simulator does not move, your scenarios stay in your checkout, and nothing here changes them.
1. Push the project
It says what it is sending, and to which host, every time.
# mutational.project.json "collector": "https://collector.mutational.in", "cloud": { "records": true } # post each sweep's full record: which check each caught fault failed through export MUTATIONAL_TOKEN=mutational_... mutational project push --project mutational.project.json
Sent: the fault catalogue, facts with each scenario's source removed, the scenario format if the project file names one, and digests of the suite and the adapter. Refused by the collector if anything sends them: scenario source, a command line, an environment, a path.
Sweep it with mutational stale --project mutational.project.json: that files each sweep under the project's name, where its page and its worklist look. A sweep made with --suite is filed under the suite file's name instead. With collector in the project file every sweep made from it is posted - an agent cannot then delete the history it is judged by - so pin the build your releases are held to (above).
2. Start a worker next to the simulator
mutational worker --cloud https://collector.mutational.in --project mutational.project.jsonIt dials out - nothing has to reach it - and runs the adapter its own project file pins. A campaign is a system's name and bounded numbers, never a command, and a worker refuses one that asks to add scenarios to its corpus or take them out.
3. Or let an agent read and queue
# a remote MCP server, with an AGENT token { "mcpServers": { "mutational": { "type": "http", "url": "https://collector.mutational.in/v1/mcp", "headers": { "Authorization": "Bearer mutational_..." } } } }
The same tools the page uses: the worklist, accept, field notes, and campaigns. An agent cannot delete anything, or replace a build's sweep with one that catches less. For an agent that writes scenarios, run mutational mcp beside the simulator, in the checkout where scenarios live.
Every command
One binary. Each takes -h for its flags, and exits 0 clean, 1 on a finding and 2 when it could not answer.
| Command | Answers |
|---|---|
mutational init | a project that runs as it is, around a toy simulator, to connect yours to |
mutational doctor | is this adapter ready to measure anything, and can the budget detect a fault? |
mutational reach | which scenarios can no fault in the catalogue touch? |
mutational stale | the sweep: what each scenario catches, fault by fault, and what decayed since the last |
mutational worker | one machine's share of a sweep fanned out across machines |
mutational gate | did this build lose coverage, asked of the hosted history; --pin holds later builds to one |
mutational trend | the whole history, sweep by sweep |
mutational blame | which build, of a list, first stopped catching it |
mutational triage | what happens if these scenarios are deleted: KEEP, DELETE or CANNOT TELL, each with its evidence |
mutational asserts | what is in the world and asserted by no scenario |
mutational catalogue | what a catalogue's faults can cause, and which hazards nothing reaches |
mutational knobs | does every setting a scenario format bounds reach the run? |
mutational generate | scenarios proposed from the format's bounds, aimed at what the corpus does not catch, kept only where a sweep says they catch something |
mutational next | what to do to the corpus next, most urgent first |
mutational accept | has a scenario that was added earned its place? From a sweep, never from its description |
mutational standin | what a rehearsal in a twin, or a judge of success, said beside what the real run did |
mutational field | what the real system did, filed beside the evidence; it raises work, and never changes a verdict |
mutational episodes, mutational reproduce | the real system's recorded episodes, and whether the simulator fails the way the robot did on one (below) |
mutational calibrate | does the simulator rank systems as the real world does? Pearson, Spearman and MMRV with bootstrap intervals, and a verdict - never from fewer than five systems |
mutational power, mutational compare | rollouts per arm to tell checkpoint B from A; and accept, reject or defer B against a margin declared first |
mutational evidence | a signed pack of what was measured, with what, and what it could not see; and its check |
mutational orient, mutational history, mutational digest | where a project stands, its sweeps one line each, and a sweep reduced to what decides the next action - for a model, in JSON |
mutational mcp | all of it to an agent over MCP; --role field only what the model operating the real system needs |
mutational ui | the same, as a page on this machine for the people in the loop |
mutational calibrate | can a result in simulation stand for one on the real system? Correlations with their intervals, and trusted, screen only, untrusted or cannot tell |
mutational power | how many trials a comparison needs, or how small a difference the ones run could have seen |
mutational compare | accept, reject or defer a new system against the old, on a Newcombe interval |
mutational evidence pack, mutational evidence verify | the evidence a release rests on, in one signed document; and the check anybody can run on it |
mutational project push, mutational project delete | send a project's catalogue and facts to the collector, never its source; take a corpus back, and with --history its sweeps |
mutational demo | the whole loop, on a known answer |
mutational licence, mutational version | which licence is in force, and what this binary is |
Episodes from the real system
An episode is whatever your adapter can read of one real run - a JSON file named ID.json in the project's episodes directory (.mutational/episodes unless the project file says otherwise). Nothing here reads inside it. describe.reproduce in the project file is a command, with {episode} replaced by the file's path, that fits your simulator to the recording, replays the system closed loop, and prints one object; it exits 0 when the run reproduces the failure, 1 when it does not, and 2 when it cannot tell. describe.episode is the command that turns an episode into a scene.
{"format": "mutational.reproduction/1", "episode": "ep-0412",
"reproduces": true, "why": "failed through grasp_held, as the robot did"}
mutational episodes --project mutational.project.json --json
mutational reproduce --project mutational.project.json --episode ep-0412Limits
Open, which is free: 100 scenarios under measurement, 1 corpus, 500 sweeps a month, 5 GB kept at rest, 5 API tokens, 365 days of history. Browser use is not metered.
Team measures up to 1,000 scenarios in 1 corpus for $180 a year; Programme 5,000 in 3 corpora for $600 a year, with campaigns run from the cloud and the collector's MCP endpoint; Enterprise is quoted. A paid plan comes with a licence, downloaded from Settings, that lets the binary sweep past 100 scenarios: point MUTATIONAL_LICENCE at it, or put it in mutational/licence.json in your user configuration directory. mutational licence says what is in force. It is checked offline; nothing is sent anywhere to check it.
A corpus is a project pushed or a suite swept in the last thirty days. Scenarios under measurement are those of every project pushed and of the latest sweep of every suite swept in the last thirty days, added up corpus by corpus; a sweep or a push that would take the workspace past its plan's corpora or scenarios is refused, and the refusal names what else is counted. A suite stops counting thirty days after its last sweep, or when it is deleted. On Team or Programme, the plan and add-ons are changed in Settings: what costs more is charged for the rest of the year, and what costs less is credited.
Calibrations and evidence packs are counted together a month - 100 on Open, 1,000 on Team, 10,000 on Programme - and count toward what a workspace keeps; a workspace registers 2 evidence keys on Open and 20 on Programme. They are kept as long as the plan keeps sweeps.
A sweep refused for a limit is not stored, and the refusal says why, and when a monthly count starts again. It is still on the machine that ran it: post it later from its recording with --replay ... --post.
Questions
Do my scenarios leave my machine?
No. Scenario source is never sent by anything here, and the collector refuses it by name. What you post is a summary and, if you ask for it, a recording stripped of what your adapter printed.
How do I delete something?
A sweep comes off its page on the dashboard, recording and all; a calibration or an evidence pack off its own; a project off its page, with its sweeps, calibrations and packs if you ask, or mutational project delete --name NAME --history. Settings removes the whole workspace. With a full token the API does the same: DELETE /v1/sweeps/<id>, DELETE /v1/calibrations/<id>, DELETE /v1/evidence-packs/<id>, DELETE /v1/projects/<name>?history=1.
Can I get a recording back?
Yes: from the sweep's page, or GET /v1/sweeps/<id>/recording with either kind of token. It is what the command line uploaded - the runs, without what your adapter printed - and mutational stale --replay reads it anywhere.
Something else?
Write to admin@mutational.in.