LeRobot vs RLDS

Both formats exist to answer the same question — what does one row of robot training data look like — and both were built by pooling datasets from many labs into one schema. The difference is which framework each one was born into: LeRobot is Hugging Face's PyTorch-native format, RLDS is Google's TensorFlow/JAX-native format, and that origin still shapes how each one feels to work with.

The short version

LeRobot: parquet tables plus encoded video

A LeRobot dataset is, at the row level, a table: each frame is a record with keys like observation.state, action, timestamp, and one observation.images.front (or similarly camera-named) key per camera — except the image keys don't hold pixels, they hold references resolved against MP4 shards. As of v3.0 (released 2026), the format moved from one parquet/MP4 file per episode to many episodes packed into shared files, with meta/episodes/ (itself chunked parquet) recording each episode's length, task, and byte/frame offsets into those shared files. The stated reason was a filesystem bottleneck: at millions of episodes, one-file-per-episode stopped scaling on both local disks and the Hub's storage backend.

Loading is unapologetically PyTorch: LeRobotDataset returns dictionaries of tensors and drops straight into a DataLoader. delta_timestamps lets you request a temporal window around a frame — useful for action chunking — without hand-rolling windowing logic. The v3.0 release also shipped StreamingLeRobotDataset, which reads shards directly from the Hugging Face Hub without a local copy, closing a capability gap against TFDS.

RLDS: episodes of steps, built for tf.data

RLDS makes the episode itself the unit of storage: a dataset is a tf.data.Dataset of episodes, and each episode wraps a tf.data.Dataset of steps plus episode-level metadata. Every step is a dictionary with a fixed key set — observation, action, reward, discount, and boundary flags is_first, is_last, is_terminal — inherited from the RL trajectory convention that TFDS and tf_agents share. observation and action are themselves nested dicts holding images, proprioceptive state, and language instructions, so the schema is flexible within a rigid outer shape.

Underneath, RLDS is TFRecord files generated with Apache Beam, which is why RLDS dataset-building scales comfortably to distributed, parallel generation jobs — Beam was built for exactly that. Consumption is native tf.data, which means it also plugs into JAX training loops via tf.data-to-numpy iterators, not just TensorFlow ones.

Where the practical differences show up

Ergonomics follow your framework. If your policy code is PyTorch — and as of 2026 most new VLA and diffusion-policy work is — LeRobot's tensor dictionaries and DataLoader integration are less friction than bridging tf.data into a PyTorch training loop. If you're in JAX or TensorFlow, the reverse is true, and RLDS's tf.data pipeline is already the shape your input pipeline wants.

Video handling diverges more than it looks. LeRobot keeps video in separate MP4 shards referenced by offset, so re-encoding video (say, changing codec or bitrate) doesn't touch the tabular data at all. RLDS embeds encoded image bytes as a feature inside the TFRecord itself, which is simpler conceptually but means image and state data are physically interleaved in the same files.

Sharding philosophy is now similar in spirit, different in mechanism. Both formats learned the same lesson — one-file-per-episode doesn't scale — and both now pack many episodes into fewer, larger files. LeRobot resolves episode boundaries through parquet metadata; RLDS resolves them through TFDS's split and shard bookkeeping.

Open X-Embodiment: the format's biggest proof point

Open X-Embodiment is the clearest evidence that RLDS scales past a single lab: it pools 1M+ real robot trajectories across 22 embodiments from 34 labs, all normalized into the RLDS episode format, and it underpins the RT-X model family. That pooling is exactly what a shared schema buys you — a training pipeline written against RLDS's step keys works unmodified against any of the 60-plus source datasets. LeRobot's Hub has since absorbed ported copies of much of that same data, so both ecosystems now have access to it, but RLDS is where it originated and where the canonical copy lives.

Converting between them

# Reading an RLDS/TFDS dataset's step structure
import tensorflow_datasets as tfds
 
ds = tfds.load("bridge_dataset", split="train")  # example OXE dataset
for episode in ds.take(1):
    for step in episode["steps"]:
        obs = step["observation"]
        action = step["action"]
        done = step["is_last"]

# Loading the LeRobot equivalent
from lerobot.datasets import LeRobotDataset
 
dataset = LeRobotDataset("lerobot/bridge_orig")  # ported copy, if available
sample = dataset[0]
state, action = sample["observation.state"], sample["action"]

Community conversion scripts walk the RLDS episode/step tree and emit LeRobot's parquet rows plus MP4 shards, and the reverse path (LeRobot to RLDS/TFDS) is a similar walk in the other direction. Both are mechanical for the standard observation/action/image fields; custom per-dataset fields need a manual mapping either way, so check field-by-field after converting rather than assuming a clean round trip.

The recommendation

Pick LeRobot if you're starting fresh in 2026 and your training code is PyTorch — which covers most current imitation-learning and VLA work, including anything built on Hugging Face's own model zoo. Pick RLDS if you're working in TensorFlow/JAX, or if your work sits close to Open X-Embodiment and the RT-X lineage where RLDS is the native format. Don't treat the choice as permanent: both formats are convertible, both now solve the same at-scale sharding problem, and a dataset recorded in one is not locked out of pipelines built on the other — it just costs a conversion pass.