The proof layer for simulation

A suite that stays green proves nothing by staying green. Mutational breaks your system on purpose and shows which scenarios notice.

Runs against

4,959 26
scenario readings to decide a deletion by hand, narrowed to a shortlist. Measured on one production AV corpus.
4 engines
MuJoCo, CARLA, Isaac Lab and a proprietary AV stack, each through its own adapter.
1 exit code
is the whole contract with your simulator. Mutational knows nothing else about your stack.

Try it here.

The real binary, compiled to WebAssembly and otherwise unchanged, with a real recording beside it: six Meta-World manipulation tasks swept in MuJoCo, 542 simulator runs.

  1. 01
    Read a real sweep542 recorded MuJoCo runs of six Meta-World tasks, replayed into the full report.
  2. 02
    The loop, on a known answerThree releases of a driving stack against one unchanged corpus. Watch a scenario go dark.
  3. 03
    What if I delete these?KEEP, DELETE or CANNOT TELL for each scenario, from what that sweep observed.
Everything else, in the playground
mutational - the real binary, running in this tab
542 recorded MuJoCo runs of six manipulation tasks. Nothing is uploaded: there is nothing to upload to.
====================================================================
MUTATIONAL REPORT
====================================================================
runner            replay: 542 recorded runs

discrimination    31.9%  (23 of 72 injected defects detected)
scenarios         6 scored, 0 skipped, 10 inconclusive defects
undeliverable     26 defects could not be expressed in the scenarios aimed at,
                  and are excluded: not exposed, and not missed either.

PER SCENARIO
? reach-v3                                   0.0%   0/10  flake 0%
  drawer-close-v3                           15.4%   2/13  flake 0%
  door-open-v3                              16.7%   2/12  flake 0%
  button-press-v3                           45.5%   5/11  flake 0%
  pick-place-v3                             50.0%   7/14  flake 0%
  push-v3                                   58.3%   7/12  flake 0%

Press "Run it here" to load the binary (3.7 MB) and produce this report
yourself, from the recording, in this tab.

From green to proven. In three steps.

01

Inject a fault.

A catalogue of ways your system can get worse, delivered through your own adapter. One exit code is the whole contract.

02

Watch what objects.

Each scenario's own pass/fail checks are the oracle. Verdicts are held against that scenario's own flake rate, so noise is never read as a catch.

  • junction_unprotected_leftKEEP
  • highway_merge_denseDELETE
  • parked_van_occlusionKEEP
  • empty_route_no_actorsDELETE
  • legacy_roundabout_v2CANNOT TELL
03

Keep what earns its place.

Keep, delete, or cannot tell - each with the evidence behind it. And it says plainly when it cannot tell.

What your corpus proves. All in one place.

Hazards

  • Collision4 scenarios
  • Bystander contactrests on 1
  • Object droppedunevidenced

Evidence by hazard

A catch counts for the check that actually failed - not for what a fault was aimed at. See which hazards rest on a single scenario.

$ mutational gate --fingerprint v3
against arm-v1

  heavy_mug_far_edge
    no longer catches  weak-grip x0.5
    no longer catches  weak-grip x0.75

exit 1 - the corpus lost coverage

A gate on lost coverage

Release after release, the gate fires on what a build stopped catching - not on a rate that moved. An unanswered question is never a pass.

Replay without a simulator

Record a sweep once where the simulator lives. Replay it anywhere into the full report - in CI, on a laptop, or in this browser.

Worklist

  • P1Bystander contact has no evidence
  • P2weak-grip is caught by nothing
  • P21 scenario set aside: harness

A worklist, most urgent first

What to do to the corpus next, for whoever - or whatever - is improving it. Field incidents raise demand; only a sweep is evidence.

POST  /v1/sweeps

curl https://collector.mutational.in/v1/sweeps
  --header 'Authorization: Bearer mutational_...'
  --json @sweep.json

API & MCP access

Everything the CLI does is an API call or an MCP tool. Post sweeps from CI, ask the gate from a pipeline, drive it all from an agent.

This tab's worker removed

fetchXMLHttpRequestWebSocketcachesEventSource+ more

Private by construction

In the browser your files never leave the tab: the worker that reads your files deletes its own network first. On the collector, adapter output is never stored.

Over the simulators you have. And the models you bring.

World models generate the universe. Simulators execute it. Mutational decides what it proves - and runs the loop that keeps a corpus worth running, from the cloud, while your simulator stays where it is.

Anyone may propose. Only a sweep accepts.

  1. 01

    The worklist

    What your corpus lacks, most urgent first - computed in the cloud by the same binary as the command line, from the sweeps you post. Field incidents raise an item. Only a sweep closes one.

  2. 02

    A model proposes

    On your own OpenRouter key, in a format your project publishes: a closed vocabulary with bounds, never a path or a command. Every proposal is a claim, and is shown as one.

  3. 03

    You approve

    Or an autopilot does, inside a budget of campaigns, scenarios and dollars that you chose. A model cannot approve what a model proposed.

  4. 04

    Your worker sweeps

    Next to the simulator, behind your VPN. It dials out, checks each scenario against the format in its own checkout, writes it to one file, and runs the adapter its own project file pins.

  5. 05

    The sweep decides

    accept answers from that sweep: earns its place, duplicate, or catches nothing. About half of what a capable model proposes tests nothing and looks fine. Those are withdrawn.

  6. 06

    A world model paints

    On your World Labs key, only the scenes that earned their place - told the kinds of thing in the scene and nothing else. A background for a camera sweep, never counted as the scene.

Built for agents. Meet Mutational MCP.

Coding agents

Any agent that speaks MCP: over stdio beside the simulator, or over HTTP to the cloud with an agent token.

Scenario engineers

A model reads the worklist, writes a scenario, and asks a sweep whether it earned its place.

Field models

The model operating the real robot gets three tools and no more: to say what happened.

.mcp.json
{
  "mcpServers": {
    "mutational": {
      "command": "mutational",
      "args": ["mcp", "--project", "mutational.project.json"]
    }
  }
}

tools  orient · next · sweep_start · job_status · accept · gate
       triage · asserts · catalogue · history · field_add · ...

One history. Every build measured against it.

suite.json

  • arm-v142.9%
  • arm-v242.9%
  • arm-v3lost 2

A shared, hosted history

Every CI build posts its sweep to one collector. Two builds line up as one series, whatever machine ran them.

API tokens

  • github actionsfull
  • scenario engineeragent

An agent's token adds. It cannot delete, or replace a sweep with one that catches less.

Tokens an agent cannot abuse

Give the model that keeps your corpus a token that cannot rewrite the history its work is gated on.

DELETE /v1/sweeps/swp_2fb1...
200  the sweep, its rows, its recording

at rest, per run:
  key · injected · outcome · reason

Yours to take back

What the adapter printed - paths, serial numbers - is stripped before upload. Delete a sweep and its recording goes with it.

Built for evidence. At every scale.

Simulation teams

A migration broke two hundred scenarios. Know which to fix, which to delete, and why.

Safety & V&V

A discrimination evidence document per release: which hazards have a test that can fail.

Robot learning

Every new policy is a new system. See what the benchmark stopped being able to catch.

Agents

A loop a model can run by itself, with a referee it cannot argue with.

Simple pricing.

API, MCP and the whole loop included. Models run on keys you bring. Free while we are in beta.

Playground

$0

No account

Look around.

  • The demo, in your browser
  • A real recording to replay
  • Every read-only command
Free in beta

Free

$0 /mo

Sign in with your email · no card

Ship with evidence.

  • 1,000 sweeps / month
  • 10 GB kept: recordings, records, pictures
  • 20 projects, a worklist for each
  • 500 proposed scenarios and 100 renders / month, on your own keys
  • 300 campaigns / month on your workers
  • The CLI for macOS and Linux
  • Hosted history and gate, 365 days kept
  • 10 API tokens, agent tokens included
  • Analyse your own files in the browser

Enterprise

Talk to us

For regulated and air-gapped teams

Run it your way.

  • A collector in your own cloud
  • Limits set by you
  • Help writing your adapter

Frequently Asked Questions

Can't find what you are looking for? Write to us.

What does Mutational actually measure?

Whether your simulation scenarios would fail if the system under test got worse. It injects faults from a catalogue - a blinded sensor, a lagging picture, a weak grip - and watches whether each scenario's own pass/fail checks object. A scenario that no fault can fail costs compute on every merge and is evidence of nothing.

What do I need to integrate it?

One script that runs a scenario and honours an exit code: 0 if it passed, 1 if it failed, 3 if the fault means nothing in that scenario, anything else if the harness broke. The fault arrives as JSON in an environment variable. It has been run through reference adapters for CARLA, Isaac Lab and Meta-World on MuJoCo, written this way.

Do my scenarios or recordings leave my machine?

Sweeps run where your simulator is, and scenario source never leaves it: the collector refuses it by name, along with your adapter's command line, its environment and any path. What --post sends is a summary - scenario names, scores, which faults were caught - and, if you ask, the recording with everything your adapter printed stripped out.

A hosted worklist needs more than a summary, and gets it only when you say so: mutational project push sends the fault catalogue, what each scenario asserts and the format a model may write in, and says what it is sending every time it runs.

In the browser, nothing leaves the tab. The worker that reads your files deletes its own network access before it accepts one, and this site's account API refuses any request larger than four kilobytes - there is nowhere here to send a file.

What are the limits of the free plan?

1,000 sweeps posted a month, 10 GB at rest - recordings, full records, pictures and project documents together - 10 API tokens, 365 days of history; 20 projects with 5 workers each, 300 campaigns, 500 proposed scenarios and 100 renders a month. Browser use is not metered. A sweep refused for a limit is still on the machine that ran it, and can be posted later from its recording.

How do I get the command-line tool?

Sign in with your email and download it from your dashboard: one static binary for macOS or Linux, on Apple silicon, Intel or Arm, with its checksum. There is an install one-liner for CI.

Can an AI agent drive it?

Yes. mutational mcp serves everything over the Model Context Protocol beside your simulator, and the collector serves the same tools over HTTP for an agent that is somewhere else. The rule both enforce is the product: a model may propose anything, and only a sweep accepts it. Give an agent an agent token and it can add to your history, ask for proposals and queue campaigns - and cannot delete, weaken, approve its own proposals, or spend your render credits.

Who pays for the models?

You do, directly, on keys you bring: OpenRouter for proposing scenarios, World Labs for renders, and optionally TypeSafe for filing field notes. A key is kept sealed - we can use it and cannot show it, to you either - and is sent only to the provider it belongs to. Every call made with one is listed in your dashboard with its host, its purpose and what kind of thing was sent. Set a spending cap at the provider: it is theirs to enforce. A set of five proposed scenarios has cost about four to eighteen cents; a draft render about eighteen.

How do I delete my data?

Delete any sweep through the API and its recording goes with it. Delete your account from the dashboard and the whole workspace goes: recordings first, then every row.

Put your tests to the test.

Sign in with your email, download the CLI, and find out what your green suite actually proves.