LeRobot vs RLDS vs ROS Bags: Picking a Robot Data Format

Every teleop session produces more than a video. A single pick-and-place demonstration on a 7-DOF arm might emit joint positions and torques at 500 Hz, wrist and scene cameras at 30–60 fps, force-torque readings at 1 kHz, gripper state, and discrete events (grasp start, contact, episode success/failure) — all on independent clocks that need reassembling into one coherent timeline. Get the storage format wrong and you either lose the ability to do that reassembly cheaply, or you lock your dataset into a single training stack.

We've moved several hundred thousand episodes through export pipelines at this point, and the honest conclusion is that there is no universal winner. Each format optimizes for a different consumer: a debugger replaying a rosbag, a PyTorch dataloader streaming shuffled batches, a TFDS pipeline doing distributed RL rollouts, an analyst running SQL over episode metadata. This is a field guide to the contenders and how we decide which one to reach for.

What a robot dataset actually contains

Before comparing containers, it helps to name the pieces they all have to hold:

• Multi-rate proprioception — joint position/velocity/effort, end-effector pose, force-torque, often at 200 Hz–1 kHz.
• Video streams — one or more cameras, typically H.264 or AV1-encoded, 480p–1080p, 15–60 fps, frequently hardware-timestamped against a PTP (IEEE 1588) grandmaster so vision and control clocks agree to sub-millisecond precision.
• Discrete events and metadata — episode boundaries, task language, success labels, operator ID, sim-vs-real flag.
• Action labels — the commanded action a policy should imitate, which for teleop data is the operator's own input, not necessarily the executed trajectory.

The hard part is timestamp alignment across streams with different rates and clock domains (controller clock, camera driver clock, host wall clock), in a way that survives being read back three years later by a library that doesn't exist yet.

The contenders

ROS 2 bag (mcap or sqlite3 storage). The default recording format for anything already living in a ROS 2 graph. As of Humble/Iron, ros2 bag record defaults to MCAP as the storage plugin, replacing the older sqlite3 backend. You get every topic, every message type, exact wall-clock and /tf timestamps, and ros2 bag play gives deterministic replay for debugging a controller regression. What you don't get is fast random access into a 40 GB bag to pull frame 812,004 without scanning — sqlite3 bags especially are slow for anything but sequential playback.

MCAP as a general container. MCAP (originally from Foxglove) is a self-describing, append-only log format: length-prefixed records, per-channel schemas, and a summary/index section at the end of the file that makes seeking cheap even over HTTP range requests. It's ROS-message-aware but not ROS-specific — you can write Protobuf, JSON, or raw byte channels into an MCAP file with no ROS runtime at all. We treat it as the right format for the raw capture layer: one MCAP per episode, everything in it, index-seekable, visualizable directly in Foxglove Studio without conversion.

LeRobot (v2 → v3). Hugging Face's LeRobot format is the closest thing to a de facto standard for imitation-learning datasets right now, given how much of the ACT/diffusion-policy/π0-style literature ships a LeRobotDataset loader. v2 stored one Parquet file and one video file per episode. v3 (mid-2025) concatenated these into Parquet shards and video files per chunk, with an episode index mapping logical episodes to byte/frame ranges — trading "one file per episode" simplicity for far better small-file and cloud-storage behavior at scale. It integrates natively with the Hugging Face Hub, so push_to_hub/from_pretrained gives you versioning and public hosting for free.

from lerobot.common.datasets.lerobot_dataset import LeRobotDataset
 
ds = LeRobotDataset("telemanual/pick-place-v1", episodes=[0, 1, 2])
sample = ds[100]
print(sample["observation.state"].shape, sample["action"].shape)
# video frames are decoded lazily and synced to the low-rate state/action rows

RLDS / TFDS. RLDS (Reinforcement Learning Datasets) is the tf.data-native episodic format built on TFRecord, used across Open X-Embodiment and most DeepMind/Google Robotics releases. It models a dataset as a sequence of episodes, each a sequence of steps with observation/action/reward/is_terminal fields, and it shines in distributed RL and JAX/TF pipelines that already assume tf.data. Its weak spot is ecosystem gravity outside TF/JAX — converting RLDS for PyTorch consumption is well-worn but nontrivial (usually via tensorflow-datasets plus a shim), whereas LeRobot assumes PyTorch from the start.

HDF5. Still common in academic robot-learning repos (it's what the original ACT and Diffusion Policy codebases shipped), because it supports typed hierarchical groups, partial reads via chunked storage, and compression in a single dependency-free file. The downsides are well known: no built-in cloud-native concurrent access, a single-writer lock that makes parallel ingestion painful, and a format finicky enough that a corrupted file is usually a fully unreadable one.

Zarr. The format we reach for when the consumer is an array-computation pipeline rather than a training loop — chunked, compressed N-D arrays with cloud object storage (S3/GCS) as a first-class backend, no single-writer bottleneck, trivial parallel read/write. It doesn't natively model "episodes" or video, so it fits derived analytics arrays (e.g., resampled joint trajectories fleet-wide) better than raw multimodal capture.

Parquet. Not an episodic robot format at all, but the right answer for the tabular slice of the problem — episode metadata, per-step scalar telemetry, success/failure labels — queryable directly from Snowflake, Databricks, or DuckDB without touching video.

Comparing them

Decision guidance

• Debugging a controller or replaying exact sensor timing → ROS 2 bag / MCAP. Nothing else preserves topic-level fidelity and plugs into rviz2/Foxglove out of the box.
• Training ACT, diffusion policies, or anything built on the current imitation-learning stack → LeRobot. The loader ecosystem and Hub integration outweigh the format's relative youth.
• Large-scale RL or JAX/TF training, especially against Open X-Embodiment-style mixtures → RLDS/TFDS.
• Fleet analytics, success-rate dashboards, dataset QA → Parquet against a lakehouse (Snowflake/Databricks), independent of how the raw frames are stored.
• Cloud-parallel preprocessing of derived arrays (resampled trajectories, computed features) → Zarr.

Format churn as a cautionary tale

The LeRobot v2-to-v3 migration is worth sitting with, because it previews a predictable failure mode: any format tied to one library's internal assumptions changes shape when those assumptions change. Teams that stored raw teleop capture only as LeRobotDataset v2 had to run a bulk conversion pass; teams that kept raw MCAP/bag alongside it just re-exported. The lesson generalizes past LeRobot — RLDS's TFRecord internals and Zarr's own v2-to-v3 spec change carry the same risk. Betting the durable copy of your data on a training-framework-specific format is a bet that the framework's schema won't move under you, and every framework's schema eventually moves.

This is why our capture pipeline records the canonical, timestamped MCAP first and treats every training format — LeRobot, RLDS, WebDataset, whatever comes next — as a materialized view generated on export, not the source of truth. When a format changes shape, we regenerate; we don't migrate.

Open formats aren't a purity position, they're a hedge: they keep the exit costs of switching training stacks close to zero, and they mean a demonstration you recorded in 2024 is still exactly as usable to a 2027 policy architecture as it was on day one.