LeRobot

LeRobot is Hugging Face's open-source library for robot learning: a dataset format (LeRobotDataset), reference implementations of the imitation-learning and vision-language-action policies most of the field trains against, and the training and evaluation CLIs that tie the two together. It's the closest thing manipulation research has to a shared PyTorch stack — a dataset published in LeRobot's format is directly consumable by any of the library's own baselines, and increasingly by third-party frameworks that adopted the same format rather than inventing another one.

How the data gets there

A Telemanual export written as LeRobotDataset is, structurally, indistinguishable from any other Hub dataset in that format: Parquet tables for low-dimensional state and action, MP4 shards for camera video, and meta/ files carrying the schema, per-feature normalization statistics, and the episode index that resolves individual episodes out of the shared shards. Loading it is one line:

from lerobot.datasets import LeRobotDataset
 
dataset = LeRobotDataset("telemanual/pick-place-v1")
sample = dataset[0]
 
print(sample["observation.state"].shape)   # proprioceptive state at this timestep
print(sample["action"].shape)              # action taken in response
print(sample["observation.images.wrist"].shape)  # decoded video frame, lazily

dataset[i] returns a dictionary of PyTorch tensors for one timestep, with video frames decoded lazily and aligned to the low-rate state and action rows — no separate video-handling code required in a training loop. delta_timestamps extends that to a temporal window around a frame, which is how action-chunking policies like ACT pull a chunk of future actions per sample without hand-rolled windowing. Task-conditioned policies pull language descriptions the same way, resolved through meta/tasks.parquet, which maps natural-language task strings to the integer task IDs stored in each episode's rows — a dataset exported with per-episode task labels already attached populates this file directly, instead of needing a separate labeling pass after the fact.

Before writing any training code, huggingface/lerobot-dataset-visualizer — a small web app maintained alongside the library — renders a dataset's episodes, camera feeds, and plotted state/action curves in a browser, which is a faster first look at a freshly exported dataset than instantiating a LeRobotDataset object and printing shapes by hand.

Training and evaluation entry points

lerobot-train is a configuration-driven CLI: point it at a dataset.repo_id and a policy.type, and it handles the rest of the training loop, checkpointing, and (optionally) logging.

lerobot-train \
  --dataset.repo_id=telemanual/pick-place-v1 \
  --policy.type=act \
  --policy.device=cuda \
  --output_dir=outputs/train/pick_place_act \
  --job_name=pick_place_act \
  --wandb.enable=true

Swapping --policy.type to diffusion, vqbet, pi0, or smolvla points the same command at a different architecture without touching the dataset side — the format is shared across all of them. For a larger job, accelerate launch --multi_gpu --num_processes=<n> $(which lerobot-train) wraps the same command for multi-GPU training.

LeRobot's own baselines aren't the only consumer. Because LeRobotDataset is a stable, documented interface rather than an internal detail of the training scripts, third-party frameworks build on it too — NVIDIA's Isaac-GR00T, for instance, trains against a LeRobot-format dataset and ships a groot policy type inside lerobot itself, though it currently expects the older v2 layout, so a v3.0 export needs the same conversion step described below before GR00T consumes it directly. That portability is a large part of why LeRobot's format functions as a shared interchange point across the current generation of manipulation policies, rather than one project's private data loader that happens to be open source.

lerobot-rollout is the counterpart for deploying and evaluating a trained checkpoint, either in simulation or against a real robot, and it supports multiple execution strategies selected with --strategy.type: base for a bare autonomous rollout, sentry for continuous recording with auto-upload, and dagger for human-in-the-loop correction recording. For slower VLA policies like Pi0 and SmolVLA, --inference.type=rtc enables Real-Time Chunking, which keeps motion smooth despite inference latency that would otherwise show up as visible stutter between action chunks.

The workflow in practice

Day to day, a team building on LeRobot moves through roughly the same loop each iteration:

1. Pull or generate a LeRobotDataset — from a Telemanual export, a self-recorded session, or an existing Hub dataset — and sanity-check dataset[0] and dataset.meta.stats before committing to a full training run.
2. Kick off lerobot-train against a baseline policy (ACT is the common first pass for a well-scoped single task) to get a working checkpoint fast, before reaching for a heavier VLA. Enable --wandb.enable=true from the first run rather than the fifth — retrofitting experiment tracking onto a training script after you've already lost track of which run produced which checkpoint is the more expensive way to learn the lesson.
3. Evaluate with lerobot-rollout, either in simulation for a fast iteration loop or on the real robot once the policy clears a simulated bar. --strategy.type=sentry or --strategy.type=dagger turn an evaluation rollout into a data-collection session at the same time, recording the human corrections that a shared-autonomy loop generates as new training data rather than throwing them away.
4. Version the dataset and the checkpoint together — a policy checkpoint is only reproducible if the exact dataset revision it trained on is still resolvable, which is why datasets and models both live as versioned repos on the Hugging Face Hub rather than as loose local files.

Gotchas

• v2 to v3 is a real migration, not a metadata bump. Datasets stored in the older per-episode LeRobotDataset v2.x layout need python -m lerobot.scripts.convert_dataset_v21_to_v30 --repo-id=<org/dataset> to reach v3.0's shared-shard layout — it rewrites the on-disk structure, not just a version string.
• The policy list moves fast. New VLAs land in lerobot on a regular cadence, and a policy that's the current recommendation in a blog post can be superseded within a few months — pin a lerobot version for any result you need to reproduce later, and check the current policy list before assuming an older tutorial's recommendation still holds.
• StreamingLeRobotDataset and LeRobotDataset are not drop-in interchangeable in every code path — some third-party training scripts written against the local, indexable LeRobotDataset need adaptation to consume the streaming variant's access pattern.
• RTC matters more than it looks for large VLAs. Skipping --inference.type=rtc on a slow policy like Pi0 or SmolVLA doesn't just cost latency — the resulting motion can be visibly jerky between action chunks, which is a real usability problem for anything contact-rich, not just a benchmark number.