LIBERO: the scene camera most tasks do not need

With pi0.5, losing the wrist camera is caught in 35 of 36 tasks. Losing the whole scene camera is caught in 11.

Published: 28 September 2026
Corpus: LIBERO (Spatial, Object, Goal, 10)
Kept by: UT Austin; pi0.5 by Physical Intelligence

pi0.5 on one LIBERO task (spatial_00, seed 1): it succeeds clean, fails without its wrist camera, and still succeeds with its scene camera black - the fault only 11 of 36 tasks caught.

Re-run for this post, seed 1 · clean: passed, as the campaign recorded · occlude 1: failed through goal_on, as the campaign recorded · blind 1: passed, as the campaign recorded

LIBERO is the benchmark nearly every vision-language-action paper reports, and it is close to saturated: pi0.5 fine-tuned on it scores 96.85% across its four suites. A score cannot say whether a task would notice a worse policy. So we degraded pi0.5 - what it sees, and what it does - and asked each task's own check.

This is the other half of what LIBERO-PRO and LIBERO-Plus measure. They perturb the world and watch success fall; we perturb the policy and watch whether the task objects.

What we ran

Tasks	40: LIBERO-Spatial, -Object, -Goal and -10, ten each, in the benchmark's order
Policy	pi0.5 (`pi05_libero`) behind openpi's policy server, unchanged
Loop	openpi's own evaluation loop, with a fault between it and the policy - and an exception counted as a broken run, not a failed task
Runs	3,856 on one rented RTX 3090, 9.1 hours, about $2

Every run's log in the recording carries LIBERO's warning that its datasets folder does not exist. That folder holds the demonstrations LIBERO trains on, and nothing here trains. An episode reads the task's BDDL file and its starting states, and those were there: a missing one stops the run, and none of the 3,856 broke. 3,592 ended in the task's own goal check; the other 264 ran their episode and the fault never came into play.

The one check a task has

A LIBERO task passes when the conjunction of its BDDL goal predicates is true at any step, and the loop stops there. 35 of the 40 tasks assert a single predicate. All ten Spatial tasks assert the same one, On(akita_black_bowl_1, plate_1). Nothing asserts that another object stayed put, that nothing was dropped after the goal was met, or how hard anything was struck.

The wrist camera is what the tasks test

Across the 36 tasks that could be scored:

Fault	Caught	Could not tell	Inconclusive
No wrist camera (occlude x1.0)	35	1	0
Half the wrist camera covered	33	1	2
No scene camera (blind x1.0)	11	8	17
Far half of the scene camera (blind x0.5)	0	24	12
The world as it was 0.6 s ago	31	0	5
... 0.3 s ago	11	3	22
... 0.1 s ago	0	27	9
Observation frozen 90% of the time	31	0	5
Half resolution	0	35	1
A fifth of the resolution	9	5	22
Pose deltas on a grid of 0.5	3	15	18
Noise of 0.1 on the motion	0	29	7
The arm moves half as far	2	18	16
Closing command weakened by 75%	0	32	2

Losing the wrist camera is caught almost everywhere. With the scene camera black for the whole episode, 8 tasks - three Spatial, four Goal, one Object - were decided as unable to tell: their failure rate was not above their own noise with nothing injected. Most of what these tasks check can be reached from the wrist view alone.

Six settings no task catches

Half the scene camera, half the resolution, 0.1 s of lag, noise of 0.1 on the motion, and a weakened grip at both strengths. Each was given to all 36 tasks.

One of those needs a caveat. Our weak grip scales the policy's closing command, and robosuite's gripper may close fully under a reduced command given time. Whether the tasks do not notice a weak grip, or our fault does not make the grip weak, we have not measured - so it is not a finding about LIBERO.

Hazards with no check at all

Five things a table-top task can cause have no check in LIBERO: acting on the wrong object, moving something the instruction did not mention, dropping a held object, striking the scene, and reaching the goal for an instant and undoing it. The last is made possible by the evaluation stopping at the first moment of success.

What this does not show

One policy, one seed, five trials a fault. pi0.5 samples, so 217 verdicts came back inconclusive; more trials would settle them.
Four tasks - goal_09, long_08, object_04, object_06 - were set aside for failing 1 of 3 runs with nothing injected.
The four suites, not LIBERO-90.
Nothing here says how good pi0.5 is. It says which tasks can tell it from a worse one.

Check it yourself

Every simulator run behind this post is published: what ran, under which fault, and whether it passed. Replaying a recording runs no simulator - the same binary reads the runs back and prints the report the figures came from. In the browser it runs in a sealed tab that cannot send anything anywhere; with the CLI it is one command.

LIBERO with pi0.5

3,856 recorded runs of 40 LIBERO tasks, replayed into the report the post's figures came from. Its totals include 217 inconclusive, 3,856 simulator runs, 40 scenarios - the figures quoted above.

Replay it in your browser runs.jsonl suite.json catalogue.json

mutational stale --replay libero.runs.jsonl \
    --suite libero.suite.json \
    --catalogue libero.catalogue.json \
    --fingerprint pi05_libero --baseline 3 --trials 5 --seed 1 --curtail=false --quiet

About these figures. Every number here comes from one sweep, recorded run by run, and each verdict is relative to one system under test and one fault catalogue - never a statement about the world. We have not yet discussed these findings with the people who keep this corpus. If you do, and something here is wrong or unfair, write to admin@mutational.in and we will correct it in place. Every run behind it is published above, and replays into these figures.