"What format should we use?" is usually three different questions wearing one trench coat: what do you record sensor data into while the robot is running, what do you keep as the long-term archival copy, and what do you actually feed to a training loop. Conflating them is how teams end up re-recording data because a training-format decision got baked into the capture pipeline, or stuck maintaining a training format that made sense for last year's model and not this year's.
Three questions, not one
Recording format is what your capture pipeline writes while sensors are streaming. It needs to be robust to a robot losing power mid-session and cheap to write at your sensor rates. Archival format is what you keep as the durable source of truth — it should outlive any specific model architecture, and often the recording format doubles as the archival format directly. Training format is what a data loader actually reads during training, optimized for random-access sampling, batching, and whatever your framework's I/O expects. These have different, sometimes conflicting priorities: a recording format optimizes for robust sequential writes, a training format optimizes for fast random reads. Trying to satisfy both with one format is why formats like a raw sqlite3 bag feel wrong for training and a training-optimized shard layout feels wrong for live recording.
A decision framework by constraint
Before comparing candidates, check which of these apply — they narrow the field faster than a feature-by-feature comparison:
- Already on a ROS 2 stack? Record in MCAP (the default storage plugin since Iron) or ROS 2 bag. This isn't really optional — it's what your drivers and tooling already speak — and see MCAP vs ROS 2 bag for which to pick between the two.
- Training in PyTorch, single institution? LeRobot's format is built for exactly this: Parquet for state and action, MP4 for video, and a growing set of policy implementations that consume it natively.
- Sharing across institutions or pooling with public datasets? RLDS/TFDS has the most precedent here — it's the format behind Open X-Embodiment, the largest cross-institution robot dataset pool to date, so compatibility with that ecosystem is a real advantage if you plan to contribute to or draw from it.
- Video-heavy at large scale, streaming from object storage? WebDataset's tar-shard layout is built for sequential I/O off cloud storage and scales past what per-file formats handle comfortably.
- Need chunked, numeric array access beyond images — point clouds, dense per-step features, large tensors read with numpy-style slicing? Zarr's chunked-array model, as used in Diffusion Policy's replay buffer, fits better than a row-oriented table.
Most teams satisfy more than one of these simultaneously, which is exactly why the record-once, convert-on-demand pattern below matters more than picking a single winner. A lab that records on ROS 2, trains primarily in PyTorch, and occasionally contributes to a shared cross-institution pool isn't choosing between MCAP, LeRobot, and RLDS — it's using all three, each for the stage of the pipeline it's actually good at, converting between them as needed rather than forcing one format to do a job it wasn't built for.
Scale changes the calculus too. A dataset of a few hundred episodes fits comfortably in HDF5 or a handful of LeRobot Parquet files with no performance concerns either way. Once a dataset grows into the tens of thousands of episodes or terabytes of video, the format's behavior under concurrent access, partial reads, and distributed training jobs starts to matter more than convenience, which is usually the point where teams migrate off a single-file format toward something chunked or sharded.
What each candidate is genuinely good at
| Format | Best for | |
|---|---|---|
| MCAP | Recording / archival | Self-describing, indexed, robust to truncation, readable outside ROS |
| ROS 2 bag (sqlite3) | Recording | Legacy default, fine for short local sessions inside a ROS workflow |
| HDF5 | Training (small–mid scale) | Single-file portability, hierarchical groups; used by robomimic and early ALOHA tooling |
| Parquet | Training (tabular state/action) | Columnar, compressed, fast filtered reads; the state/action half of LeRobot |
| Zarr | Training (chunked arrays) | Chunked N-dimensional arrays with parallel access; used by Diffusion Policy's replay buffer |
| LeRobot | Training (PyTorch ecosystem) | Parquet + MP4, Hugging Face Hub-native streaming as of v3.0 |
| RLDS / TFDS | Training (TensorFlow, cross-institution) | Standardized episode/step schema; backbone of Open X-Embodiment |
| WebDataset | Training (large-scale streaming) | POSIX tar shards, sequential I/O, scales cleanly off object storage |
A few of these deserve a specific caveat. HDF5 stores an entire dataset as one hierarchical file, which makes it easy to hand around but weaker under concurrent writes and parallel reads at large scale than formats built around many independent chunks — see HDF5 vs Parquet for the detail. LeRobot's format changed meaningfully at v3.0: earlier versions stored one Parquet and one video file per episode, while v3.0 packs multiple episodes into shared Parquet/MP4 files with relational metadata resolving episode boundaries, cutting filesystem overhead and enabling direct streaming from the Hugging Face Hub without a full local download. RLDS organizes a dataset as episodes of steps, each step a dictionary of observation, action, reward, and boundary flags (is_first, is_last, is_terminal) — a schema TFDS indexes and any tfds.load() caller can consume, which is exactly the property that makes it work for pooling datasets across institutions that don't otherwise agree on anything else.
The record-once, convert-on-demand pattern
The pattern that avoids re-collecting data every time a training format falls out of favor: record and archive in a lossless, general-purpose format — MCAP is the strongest default — and treat every training format as a view generated from that archive rather than the primary copy. When a new model architecture wants its data as RLDS instead of LeRobot, or a collaborator needs WebDataset shards, that's a conversion job against the archive, not a new collection campaign.
- Raw sensor truth never gets lossy-transformed away by an early training-format choice.
- Switching model architectures or frameworks doesn't require re-collecting episodes.
- One archive can feed multiple simultaneous training pipelines without duplication of the source data.
- Schema and calibration metadata travel with the archive, not a separately-maintained training copy.
- You maintain conversion scripts for each training format you actually use.
- Storage cost for the archival copy plus at least one training-format copy.
- Conversion correctness — especially timestamp handling — has to be verified once per target format, not assumed.
That last point in the cons list is not a footnote: a conversion step is exactly where synchronized timestamps quietly get collapsed into arrival-time noise if the pipeline isn't explicit about which timestamp it propagates — the same failure mode covered in time syncing sensors onto one clock.
Putting it together
Start from recording, not from training. If you're on ROS 2, record MCAP — that decision is nearly free and it keeps every future option open. Pick a training format based on your actual framework and sharing needs today, knowing it's a conversion target rather than a commitment, and revisit it when your dataset size, sharing requirements, or model architecture changes enough to justify the conversion effort. The teams that end up stuck aren't the ones who picked the "wrong" training format — they're the ones who recorded directly into one and lost the archival copy that would have made switching cheap. For a closer look at how these tradeoffs played out across several hundred thousand real episodes, see LeRobot vs RLDS vs ROS bags.