LeRobotDataset is Hugging Face's dataset format for robot learning, and it is close to the de facto default for the imitation-learning literature that ships a PyTorch dataloader — ACT, Diffusion Policy, and most π0-style work load through a LeRobotDataset object. Where RLDS commits to TF/JAX, LeRobotDataset commits to PyTorch and the Hugging Face Hub, and its layout has been redesigned once already to hold up at larger scale.
The core design: decouple storage from the API
The organizing idea of the current layout, v3.0, is separating how data is physically stored from how a training script accesses it. Physically, the dataset is a small number of large files. Logically, dataset[100] still returns one training sample — a dictionary of tensors for a single timestep of a single episode — with the translation between the two handled by metadata rather than by file boundaries.
That distinction exists because the previous layout, v2.x, stored one Parquet file and one video file per episode. That's simple to reason about, but a dataset with a few hundred thousand episodes becomes a few hundred thousand small files, which is slow to enumerate, slow to open on first load, and unfriendly to object storage. v3.0 concatenates many episodes into shared Parquet and MP4 shards and reconstructs episode-level views from offsets recorded in metadata.
The three pillars
- Tabular data
- Low-dimensional, high-frequency signals — state, action, timestamps — in Apache Parquet under data/, chunked with many episodes per file
- Visual data
- Camera frames concatenated and encoded to MP4 under videos/, sharded per camera
- Metadata
- JSON/Parquet under meta/ — schema, frame rate, normalization stats, and per-episode offsets into the shared shards
The meta/ directory carries the pieces that make the other two pillars interpretable:
meta/info.json— the canonical schema: feature names, shapes and dtypes, frame rate, the dataset's codebase version, and path templates for locating shards.meta/stats.json— per-feature mean, std, min, and max, used for input normalization at training time.meta/tasks.parquet— natural-language task descriptions mapped to integer task IDs, for task-conditioned policies.meta/episodes/— chunked Parquet records of per-episode length, task, and byte/frame offsets into the shared data and video shards — the index that turns a shard back into individually addressable episodes.
Why decoupling video from tabular data matters
State and action arrays are small per timestep but read constantly — a training step touches every feature in the batch. Video frames are large but usually only a subset is needed per batch (often one or a few camera views, sometimes a short temporal window via delta_timestamps). Mixing both into one row-oriented structure forces every reader to pay video-decoding cost even for batches that only need proprioception, and forces every video reader to fight for the same file locks as tabular access.
Splitting them lets each half use the storage format suited to it: Parquet's columnar layout and memory-mapping benefit the small, dense tabular data. Video codecs (H.264/AV1) benefit the camera streams, cutting storage by an order of magnitude relative to storing raw frames as arrays — the same trade-off HDF5-based formats historically avoided by storing raw arrays and paying for it in disk size.
Hub distribution and consumption
A LeRobotDataset is designed to be pushed to and pulled from the Hugging Face Hub like any other Hub dataset — versioned, publicly hostable, and loadable by repo ID with no separate download step:
from lerobot.datasets import LeRobotDataset
dataset = LeRobotDataset("telemanual/pick-place-v1")
sample = dataset[100]
print(sample["observation.state"].shape, sample["action"].shape)
# video frames decode lazily and align to the low-rate state/action rowsFor datasets too large to download locally, StreamingLeRobotDataset reads directly from the Hub (or from an HF storage bucket) without materializing the dataset on disk first — the counterpart to RLDS's sharded-TFRecord streaming, built for the same reason: multi-terabyte mixtures that no single worker should have to copy locally.
LeRobotDataset objects return plain PyTorch tensors from __getitem__, so they plug directly into torch.utils.data.DataLoader. Beyond the lerobot library's own training scripts, third-party frameworks build against the format too — NVIDIA's Isaac-GR00T trains on LeRobot-format data and ships a groot policy type inside lerobot itself, though GR00T currently expects the older v2 layout rather than v3.0's shared shards, so a v3.0 export needs the same conversion pass described below first. That LeRobot functions as a shared interchange point at all across the current generation of manipulation policies, rather than a training script's private data loader, is what makes cases like this worth the conversion step.
Because the format and the training scripts share a maintainer, features land in both places together — a new observation key, a new normalization stat, or a new video codec shows up in LeRobotDataset and in lerobot-train's baselines in the same release, rather than one project chasing another's format changes after the fact. Training runs against it are typically logged to Weights & Biases through the same CLI flags used for ACT and Diffusion Policy baselines, so a dataset swap doesn't require re-plumbing experiment tracking.