This post is less about the numbers and more about how we got there: the bet we made, the things that broke, and what we believe now that we did not believe a year ago.

The bet

The project started with an observation that is hard to unsee. Used directly as a robot policy, GPT-6 Astra ranks above every public entry on the RoboDojo leaderboard. Given a coding harness, it solves 94.0% of LIBERO-PRO episodes. None of these models were trained to output motor commands. What they do is write programs: look at the scene, call a segmenter, measure a distance, move the gripper under a force limit, check the result, try again.

The way we read this: a large vision-language model already has a surprisingly good model of space. It knows that the bowl is left of the plate, that a drawer opens along its handle's normal, that a cube in a closed gripper moves with the gripper. What it lacks is a channel through which that knowledge can reach the motors. Code is such a channel. It turns spatial understanding into explicit, composable, checkable operations, and it does so in the representation the model is best at: language.

Compare the two dominant alternatives.

• Vision-language-action models ask the network to predict continuous, high-dimensional action trajectories directly. This is a hard regression problem, it is tied to one embodiment's action space, and the data it needs (teleoperated demonstrations) does not scale the way text and images do. Every new robot is, to a first approximation, a new dataset.
• World-action models predict future frames and actions jointly. Pixels are an expensive and indirect proxy for physics: two frames can look nearly identical while the contact state, the one thing that matters for a grasp, is completely different. Learning physical law through video prediction is possible, but it is learning it the long way round.

Language is different. It is a dense, compositional representation that already encodes most of the relations a manipulation task needs, and a program built from it is grounded at every step by the robot's own feedback. Our hot take: for manipulation that requires reasoning, code is the right action space, and the hard part is not the representation but training a small model to use it.

That last part is the problem. Every strong code-as-policy (CaP) agent today runs on a closed frontier model. You cannot fine-tune it, you cannot put it on the robot, and at \$8–22 per episode for the best open-harness systems you cannot run it at scale. An off-the-shelf 4B model placed in the same harness solves nothing: 0.0% on LIBERO. So the question became: can we post-train a small open model into a CaP agent, the same way coding and tool-use agents are post-trained, and what does it take?

Decision 1: no skills, no VLA

The first decision was what the agent is allowed to call. It is tempting to give it grasp(), place(), open_drawer(), or a trained VLA as a skill. Benchmarks get much easier. But each of these moves the hard part of the problem out of the policy and into hand-written code or a separately trained network that has to be rebuilt for every robot. A VLA as a skill also brings back exactly the scaling problem we were trying to avoid.

So CaPtain-Harness exposes 22 low-level capabilities and nothing else: perceive (observe, segment, point, mark_point, track_target), measure (object and contact geometry, observation comparisons), act (move_to, hold, recover, release), and check (motion previews, robot state). Down to IK-level motion control, every tool is a primitive. A grasp is something the policy has to compose: measure the object, choose an approach, descend under a force guard, close, verify.

The second design choice was the receipt. Robot calls never raise. Each returns a structured result, e.g. "stopped: contact · moved 0.12 m · peak 6.2 N · jaw 8 → 1 cm". A small model cannot recover from a failure it cannot see, and, as it turned out, much of what RL improves is how the policy reads these receipts.

The payoff of this decision came later: the same API runs on LIBERO, RoboTwin 2.0, RoboDojo, SimplerEnv, CaP-Bench and a real robot. Only the backend underneath changes.

Here is a piece of an episode the trained 4B model wrote (verbatim code, receipts shortened):

# request 6 (after the grasp): lift, re-observe, re-identify bowl and plate, measure
lift = go(np.array(obs2["value"]["robot_state"]["end_effector_position_m"]) + [0, 0, .12], q,
          speed=0.035, mode="manipulation")
obs3 = robot.observe()
b2 = robot.mark_point(obs3["value"]["observation_handle"], "agentview", .75, .421, label="held black bowl")
p2 = robot.mark_point(obs3["value"]["observation_handle"], "agentview", .78, .725, label="plate")
g2 = robot.measure_object_geometry(b2["value"]["detection_id"], p2["value"]["detection_id"])
>>> go: True none bounded Cartesian sweep reached its endpoint

# request 7: carry the bowl by the measured object-to-destination offset
transfer = np.array(obs3["value"]["robot_state"]["end_effector_position_m"])
transfer[:2] -= np.array(g2["value"]["object_to_destination_xy_offset_m"])
trans = go(transfer, q, mode="manipulation")
>>> move_to: ok; stopped_reason=target_reached executed_distance_m=0.1681 max_observed_force_n=5.198

# request 13: fresh final measurement, then finish
>>> {'object_to_destination_xy_offset_m': [0.0012, 0.0170], 'xy_distance_m': 0.0171, ...}
"The bowl is resting on the plate. Fresh measurements show a center offset of about 1.7 cm,
 within the required 2 cm tolerance."  -> robot.finish(...)

Nothing here is a skill. The model clicks pixels to ground objects, measures, moves by the measured offset, and checks the final relation before it declares success. This is what we mean by letting spatial understanding reach the motors.

Lesson 1: the model stopped looking

SFT on synthetic trajectories, generated by GPT-6 Astra in the same harness, gets a small model off the ground. Our first SFT models looked fine on held-out states. They were quietly broken.

We noticed when we replaced the input image with a different scene and the likelihood of the predicted pixel coordinates did not change. The model was not reading the image. It had learned to output coordinates from the task identity alone. The cause was the data: most episodes started from LIBERO's official initial states, where each object sits in nearly the same place every time (per-object position standard deviation about 1.4 cm). Under that distribution, memorising the layout is a perfectly good solution, and gradient descent found it.

The fix was in the data, not the model. We regenerated data from wide-layout start states (standard deviation about 15 cm), mixed all task suites, randomised lighting, camera pose and materials during RL, and masked earlier mark_point coordinates in the history with probability 0.2, so every new call has to be grounded in the current image. That single change is also why the policy later works on a real robot.

The general point is worth stating plainly: visual grounding in a fine-tuned VLM is fragile. It is inherited from pretraining, and it is erased by any data distribution in which the image is not needed to predict the label. For embodied SFT, layout diversity matters more than data volume.

Lesson 2: RL that goes nowhere

With a working SFT model, we started RL. It did not learn. Success rates drifted up and down within noise, and nothing in the pipeline was wrong.

The diagnosis came from looking at the updates rather than the rewards. Consecutive updates were nearly orthogonal, and the distance of the weights from the start grew like the square root of the number of steps. The policy was doing a random walk.

The reason is batch size. We measured the gradient-noise scale \(B_{\text{noise}}\) of the 4B policy: about 9,500 episodes (95% lower bound 3,300). An update from 256 episodes is mostly noise. Accumulating over \(K = 4\) rounds, 1,024 episodes per optimizer step, fixed it. In hindsight this is obvious; multi-turn episodes with one binary outcome are about the noisiest training signal there is. If we could give one piece of advice to anyone doing RL on multi-turn agents: measure \(B_{\text{noise}}\) before you tune anything else.

Lesson 3: credit lives where physics changes

Bigger batches make the gradient point somewhere; they do not make it point at the right requests. A LIBERO episode has 20–60 requests and one outcome. Most requests look at the scene or measure something. A handful, the grasp, the carry, the release, decide the episode. An outcome reward spreads the same signal over all of them.

The insight that unlocked this was simple: the simulator already knows what the policy is trying to infer. It knows every object pose, whether something is in the gripper, and how many sub-goals are satisfied. A policy should not see this, because it will not have it at test time. But a critic can.

So we train a small critic, an ensemble of five MLPs over 36 physical features, to predict the probability of success from the simulator's state, and use the change in that prediction as each request's credit:

accumulated with GAE (\(\lambda = 0.9\)) and combined with a group-relative outcome term. The critic is co-trained online, as in SAO, and disappears at test time.

Physical state only changes through physical action. A request that only perceives, however many tokens it produces, gets almost no one-step credit, and receives its share through GAE from the action it enables. In an offline study on 15k episodes, the critic predicts final success with AUC 0.970. Successful grasps get \(\Delta V = +0.058\), failed grasps \(-0.0075\), perception-only requests \(-0.0002\). 65% of the advantage mass goes to moves, 14% to grasps, 14% to releases, and 1.3% to perception. This is what credit assignment should look like.

Two things we got wrong first, both instructive:

• Time leaks into credit. The critic needs the request index \(t\) to be calibrated: half a task done at request 55 is worse than at request 5. But if \(t\) advances inside the difference, the passage of time alone produces credit. We hold \(t\) fixed within each difference.
• Anything about the policy is a clock. Adding counters of the policy's own calls made the critic slightly more accurate and pulled credit onto perception requests. The policy's LM hidden states were worse: they decode the request index with \(R^2 = 0.80\), and used as critic inputs they moved 35% of the advantage mass onto perception. The critic should see the world, not the policy.

Lesson 4: knowing when to stop

We report two numbers throughout. Host success is the benchmark's goal predicate on the final state. Strict success additionally requires the agent to call finish() itself. On a robot, only the second counts: an agent that never says it is done cannot be deployed.

Late in training the two come apart. After about round 340, host success holds while strict success falls from about 60% to 44%. The policy still solves the task; it keeps re-measuring a scene that is already solved until it runs out of requests. The reason is structural: the critic is trained on host success, so every post-success state has \(V \approx 1\), and the decision to stop gets no credit from it at all. We use the round-300 checkpoint as CaPtain-4B. The real fix is to give termination its own target, and it is the first thing we will change.

Where it landed

• LIBERO (held-out initial states): base 0.0% → SFT 55.4% → RL 73.8%.
• Perturbations: RL adds +19.0 on LIBERO-PRO Pos+Task (to 78.5), +17.6 on LIBERO-Plus, +21.2 and +18.8 on LIBERO-MAX Dynamic and Base. The gains are not memorisation of training layouts.
• New tasks and embodiments, no further training: RoboTwin 2.0 (official Hard protocol) 22.6%, above X-VLA (20.9%) and Xiaomi-Robotics-0 (18.2%); SimplerEnv WidowX 83.3%; CaP-Bench 75.0, above CaP-X's multi-turn agents on Gemini-3-Pro (73.5) and GPT-5.2 (55.5); RoboDojo Score 1.7.
• A real arm: after adaptation on episodes from a coarse simulation of an XR1 robot, with no real-world data, the same API and receipts drive the real arm.

The transfer results are the part we find most telling. The policy was trained with RL on 40 LIBERO tasks with one arm. It runs on new robots because nothing it learned is tied to an action space: it learned to look, measure, move and check. RoboDojo marks the next frontier: tasks that need two arms at once, handovers or memory.

What we believe now

1. Code is an underrated action space. A VLM's spatial understanding is real but locked behind its output format. Code unlocks it, and the resulting policy is portable across embodiments for free, because the embodiment lives below the API.
2. Robot agents should be post-trained like every other agent. Fixed harness, SFT for a cold start, RL for the rest. There is nothing special about manipulation that requires a different recipe; there are just different failure modes.
3. The interface is the ceiling. What the API cannot express, the policy cannot learn. Sustained pressing (CaP-Bench Wipe) is impossible with guarded motion primitives, no matter how good the model. Designing the primitives is designing the hypothesis space.
4. Diversity beats volume, and grounding is fragile. A fine-tuned VLM will stop looking at the image the moment the image stops being necessary.
5. Multi-turn RL is a batch-size problem before it is an algorithm problem. Measure the noise scale first.
6. Privileged information belongs in the critic. The simulator is not only an environment; it is a teacher of credit. The policy stays deployable because it never sees what the critic sees.

Open questions

• Termination. Stopping needs its own credit target; host success is the wrong label for the decision to finish.
• Two arms. A critic over joint two-arm state is the obvious next step.
• Off the simulator. The critic needs object poses a real robot does not have. With 2 cm pose noise, an estimated-state critic agrees with the simulator critic at only 0.49 correlation, so RL on hardware needs better state estimation.
• More training worlds. RL saw only LIBERO. How far does transfer go when RL itself sees many embodiments?

Release

Everything is open under MIT: the training code, CaPtain-Harness with adapters for every benchmark above, the SFT data, and the CaPtain-4B and CaPtain-4B-SFT weights. Train your own model on our pipeline, or test yours in our harness. The video walks through the whole pipeline.

References

1. Liang et al. Code as Policies: Language Model Programs for Embodied Control. ICRA 2023.
2. Liu et al. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. NeurIPS 2023.
3. Zhou et al. LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization. 2025.
4. Fei et al. LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models. 2025.
5. Zhang et al. LIBERO-MAX: Do Robot Policies Adapt When the World Changes? 2026.
6. Chen et al. RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation. 2025.
7. Chen et al. RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies. 2026.
8. Fu et al. CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation. 2026.
9. Hou et al. Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning (SAO). 2026.
10. Pinto et al. Asymmetric Actor Critic for Image-Based Robot Learning. RSS 2018.
11. Schulman et al. High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR 2016.
12. McCandlish et al. An Empirical Model of Large-Batch Training. 2018.

Citation

@misc{fan2026captain,
  title  = {CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents},
  author = {Fan, Yijia and He, Xilin and Jiang, Yilei and Wu, Dongming and Wang, Jiaxu and Yue, Xiangyu},
  year   = {2026},
  note   = {Blog post}
}