A vision-language-action model is a single network that takes in camera images and a natural-language instruction and outputs robot actions, collapsing what used to be a pipeline of separate perception, language, and control modules into one forward pass. The defining architectural choice is where it starts from: not a randomly initialized network trained only on robot data, but a vision-language model (VLM) already pretrained on internet-scale image-text pairs, adapted to speak actions instead of, or in addition to, words.
The lineage: from RT-1 to today's generalist policies
- RT-1 (Google, Robotics Transformer for Real-World Control at Scale, RSS 2023) tokenized camera images and language instructions and predicted discretized robot actions with a transformer, trained on 130k episodes across 700+ tasks collected from 13 robots over 17 months. It was not built on a pretrained VLM, but it established the tokenized-action recipe later VLAs reused.
- RT-2 (Google DeepMind, Vision-Language-Action Models Transfer Web Knowledge to Robotic Control, 2023) is generally credited with the term: it co-fine-tunes a pretrained VLM — PaLI-X (55B parameters) or PaLM-E (12B) — on robot trajectories alongside web-scale vision-language data, representing actions as text tokens the same architecture already predicts.
- Octo (UC Berkeley, Stanford, Carnegie Mellon, and Google DeepMind, 2024) is an open-source generalist transformer policy pretrained on 800k episodes drawn from Open X-Embodiment, released in Octo-Small (27M parameters) and Octo-Base (93M) sizes, with a diffusion head for action prediction and support for finetuning to new robot embodiments.
- OpenVLA (Stanford, UC Berkeley, Toyota Research Institute, Google DeepMind, Physical Intelligence, and MIT, June 2024) is a fully open 7B-parameter VLA built on a Prismatic VLM — a fused DINOv2 and SigLIP vision encoder feeding a Llama-2 language model — trained on 970k trajectories from Open X-Embodiment and predicting discretized 7-DoF actions.
- π0 (Physical Intelligence, October 2024) pairs a PaliGemma (3B) vision-language backbone with a 300M-parameter flow-matching "action expert" (3.3B parameters total) that predicts chunks of up to 50 future actions, aimed at general-purpose control across multiple robot embodiments rather than a single platform.
Why a VLM backbone
Robot demonstration datasets, even pooled across dozens of embodiments, are minuscule next to the image-text corpora VLMs pretrain on. Building on a pretrained VLM lets a policy inherit priors it would never learn from robot data alone — what a spatula looks like under different lighting, what "next to the red block" refers to, common associations between objects and how they're used. RT-2's central empirical claim was that this transfer is real: co-fine-tuning on web data alongside robot data produced measurably better generalization to novel objects and instructions than training the same architecture on robot data by itself.
This is also why the field converged on adapting existing VLMs rather than pretraining vision-language understanding from scratch on robot data. A VLM like PaLI-X or PaliGemma has already seen orders of magnitude more images, captions, and object references than any robot dataset will ever contain, and re-deriving that visual and linguistic grounding from robot demonstrations alone would need a robot dataset that does not exist and would be prohibitively expensive to collect. Adapting a pretrained backbone turns a data problem the field cannot solve into a fine-tuning problem it can.
Turning language-model outputs into robot actions
A language model predicts tokens from a fixed vocabulary; a robot needs continuous joint or end-effector commands. Two strategies bridge that gap. Discretized-token models — RT-2, OpenVLA — bin each dimension of the action space and map the bins onto otherwise-unused tokens in the model's vocabulary, so the transformer predicts an action the same way it predicts a word, autoregressively. This reuses the architecture unchanged but caps resolution to the bin width and predicts one token at a time. Continuous-action models — Octo's diffusion head, π0's flow-matching action expert — attach a separate module that consumes the VLM's features but outputs a chunk of continuous actions in one pass, trading architectural simplicity for smoother, higher-resolution, faster control. The underlying issue — a single deterministic prediction cannot represent the genuinely multiple valid ways a demonstration could continue — is the same multimodality problem covered under behavior cloning, and the same fixes (action chunking, diffusion) apply.
Cross-embodiment training and what Open X-Embodiment changed
Before 2023, robot learning papers trained and evaluated on their own single-robot dataset, and results rarely transferred to a different arm or gripper. The Open X-Embodiment collaboration — 21 institutions pooling demonstrations from 22 robot embodiments into one dataset, described in Open X-Embodiment: Robotic Learning Datasets and RT-X Models (2023) — showed that a single policy trained across all of them generalized across robot form factors better than policies trained on any one robot's own data. Octo, OpenVLA, and π0 all pretrain on Open X-Embodiment or its derivatives, which is why cross-embodiment pretraining is now the default starting point for a generalist policy rather than a research curiosity.
Data format implications
Pooling recordings from dozens of robots and labs only works if the data describes its own structure instead of assuming one hardcoded schema per robot. RLDS — Google's Reinforcement Learning Datasets specification (2021), which represents an episode as a sequence of steps each carrying observation, action, and metadata — is the format Open X-Embodiment standardized on, and it's built on top of TensorFlow Datasets. LeRobotDataset, Hugging Face's more recent format, covers similar ground with Parquet files for state and action data and MP4 shards for camera streams, designed for streaming large datasets directly off the Hub rather than downloading them whole first. Which of the two a team should record to, and why, is covered in choosing a robot data format.