Most of what determines whether a fine-tuned VLA works on your robot happens before the first training step: how many episodes you have, whether the action space is actually consistent across them, and whether your normalization and splits are computed correctly. The model architecture matters far less than teams expect. This guide walks the path in the order the decisions actually arrive, from data audit to the first real-robot rollout.
Audit your data before you train
Before touching a training script, answer three questions honestly.
Episode count, against the right baseline. OpenVLA is a 7B-parameter model trained on roughly 970k real-world demonstrations from the Open X-Embodiment mixture; Octo is a smaller 93M-parameter transformer with a diffusion action head trained on about 800k OXE episodes. Neither number is the bar you need to clear — both were trained once, broadly, so that you don't have to repeat that scale of collection. Fine-tuning on a new task typically works with a small target dataset; training a comparable generalist policy from scratch does not.
Action space consistency. Every episode in a dataset must express actions in the same representation — joint positions vs. joint velocities vs. end-effector pose deltas, absolute vs. relative, the same units and control frequency. A dataset assembled from multiple recording sessions, operators, or even software versions can drift here without anyone noticing until training loss looks fine and rollouts don't work. Spot-check action ranges per episode, not just in aggregate.
Instruction coverage. A VLA is language-conditioned; every episode needs a short, accurate natural-language instruction if you want language conditioning to do anything useful. Retrofitting instructions after the fact is expensive labeling work — see language annotation for robot data if yours are missing.
Choose fine-tune or train from scratch
For almost every team adding a new task or a new robot to an existing embodiment class, fine-tuning a pretrained open policy is the right call.
- Your robot's action space resembles what OpenVLA or Octo were trained on (joint or end-effector control, parallel-jaw or similar grippers).
- You have a small-to-moderate target dataset for the new task and want visual grounding and manipulation priors for free.
- You want a working baseline in days, not months of compute.
- You're adding tasks or objects to a robot family the pretraining mixture already covers.
- Your sensing or action space is far outside any open pretraining mixture — a novel end-effector, an unusual control abstraction, or a non-visual primary modality.
- You have hundreds of thousands of episodes across many tasks and the compute budget to match.
- You need full control over the pretraining data mixture for a research question, not just a working policy.
LeRobot's own training stack ships baselines beyond VLAs worth knowing about too — ACT (Action Chunking Transformer) and Diffusion Policy are smaller, faster-to-train imitation-learning architectures that don't carry a pretrained language-vision backbone at all. They're often the pragmatic choice for a single well-scoped task where you don't need open-vocabulary language conditioning.
Convert your recordings into a training format
Whatever you fine-tune, the framework wants a specific on-disk layout, not your raw capture. The two dominant options are LeRobotDataset (PyTorch-native, Hugging Face Hub-distributed, consumed by the lerobot library's ACT, Diffusion Policy, and VLA training scripts) and RLDS (TFDS-based, the format Open X-Embodiment and Octo standardized on). See LeRobot vs RLDS if you haven't picked one — the short version is PyTorch pipelines want LeRobot, TF/JAX pipelines want RLDS, and converting between them later is real engineering work, not a metadata edit.
If your raw capture is MCAP or a ROS 2 bag, conversion is a scripted export, not a rewrite from scratch:
from lerobot.datasets import LeRobotDataset
dataset = LeRobotDataset.create(
repo_id="your-org/pick-place-v1",
fps=30,
features={
"observation.images.wrist": {"dtype": "video", "shape": (480, 640, 3)},
"observation.state": {"dtype": "float32", "shape": (7,)},
"action": {"dtype": "float32", "shape": (7,)},
},
)
for episode in raw_episodes:
for frame in episode.frames:
dataset.add_frame(frame)
dataset.save_episode(task=episode.instruction)
dataset.finalize() # required before push_to_hub — writes the Parquet footerKeep the lossless raw capture too. Export is a derived artifact; if the training format's internals change again — as LeRobot's did between v2 and v3 — you re-export instead of losing history.
Compute normalization statistics from your own data
Policies are trained on normalized inputs and outputs — typically z-scored or min-max scaled actions and proprioceptive state. Those statistics have to come from your dataset. meta/stats.json in a LeRobotDataset (or the equivalent step in an RLDS pipeline) holds per-feature mean, std, min, and max computed over your episodes, and that's what the training script reads at startup.
Reusing a pretrained checkpoint's normalization — copying stats.json from someone else's dataset, or worse, hardcoding constants from a paper — silently rescales your action space. The failure is quiet: training loss looks normal, because the network learns to compensate for a fixed offset, but the compensation doesn't transfer to real joint commands at inference. Always regenerate normalization from your own recordings before a fine-tuning run.
Split train and validation sets by episode
Hold out whole episodes for validation, never individual frames. Splitting by frame lets adjacent timesteps from the same trajectory land in both sets — the model effectively memorizes a few frames of context around a val frame it has already seen elsewhere in the same episode, and validation loss stops meaning anything. A common split is 85–90% of episodes for training and the remainder held out, chosen with enough variation (not just the last N episodes recorded, which tend to cluster in time and operator).
On genuinely small datasets — a few dozen episodes for a single new task — a formal validation split can be too small to be informative at all. Some teams skip it and lean entirely on real-robot rollouts as the evaluation signal, which is defensible as long as you're honest that you're flying without a loss-based sanity check.
Train, and treat validation loss as a sanity check
Validation loss is useful for exactly one thing during VLA fine-tuning: catching a training run that has clearly gone wrong — diverged, overfit hard, or stopped learning. It is a poor proxy for what you actually care about, which is task success on the robot. Behavior-cloning losses measure how well the policy reproduces the demonstrator's actions at each timestep; they don't measure whether small per-step errors compound into a failed rollout, which is exactly the failure mode that dominates real deployments. Don't pick a deployment checkpoint by validation loss alone.
Evaluate on the real robot and iterate on data
Run the policy on hardware, on the actual task, with held-out scene variation the model hasn't seen. Log failures by phase — approach, grasp, transport, place, release — because the phase where rollouts break tells you what to fix, and it's almost always a data problem, not a hyperparameter one. Missing coverage of an object pose, an under-represented lighting condition, or a mislabeled subset of episodes will show up as a specific, repeatable failure mode long before a learning-rate sweep changes anything.
This is the loop that actually moves fine-tuning results: evaluate on the robot, categorize the failure, collect or fix the data that addresses it specifically, retrain, and evaluate again. Teams that treat data collection as a one-time step before training, rather than an ongoing response to what the robot is getting wrong, plateau early — the fix is almost never a bigger model or a longer sweep. See scaling robot data collection for how to structure that loop as collection volume grows past what one person can review by hand, and keep a written log of which failure mode each new batch of data was collected to address, so the next reviewer can tell whether it worked.