HDF5 vs Parquet for Robot Data

Both formats solve "how do I store a lot of structured data efficiently," and both show up somewhere in most robot data pipelines — but they were built for different access patterns, and using the wrong one for a given piece of data shows up later as either slow reads or an awkward schema.

The short version

What HDF5 does well

HDF5's unit is the dataset: a typed, chunked, optionally compressed N-dimensional array, addressable by a path inside a hierarchical namespace of groups — conceptually a filesystem inside a file. Chunking is the important part for robot data: you choose a chunk shape (say, one chunk per timestep of a [T, H, W, C] image stack), and a reader can pull one chunk without touching the rest of the array. That makes HDF5 genuinely good at "give me timesteps 4000–4050 of this 200,000-timestep tensor" without a full-file scan.

import h5py
import numpy as np
 
with h5py.File("episode_014.h5", "w") as f:
    # chunked per-timestep, compressed
    f.create_dataset(
        "observations/depth",
        shape=(2000, 480, 640),
        dtype="f4",
        chunks=(1, 480, 640),
        compression="gzip",
        compression_opts=4,
    )
    f["observations/depth"][100] = np.zeros((480, 640), dtype="f4")
 
with h5py.File("episode_014.h5", "r") as f:
    frame = f["observations/depth"][100]  # reads one chunk, not the whole array

The cost is that HDF5 was designed around a local, mostly-POSIX filesystem model. Concurrent writers are awkward (single-writer/multiple-reader mode exists but is limited), and while cloud-object-store drivers exist, HDF5's chunk-index-then-seek access pattern maps less cleanly onto HTTP range requests than Parquet's footer-plus-row-group layout does. A corrupted or truncated HDF5 file can also be harder to partially recover than an append-only log format.

What Parquet does well

Parquet's unit is the column. Data is laid out as row groups, and within a row group, one contiguous chunk per column — so a query that only needs three of forty columns reads only those three, and a query filtered on a partition key can skip whole row groups using the file's statistics without decompressing anything. That's the columnar-analytics access pattern: scan a lot of rows, touch few columns, filter aggressively before reading.

import pyarrow as pa
import pyarrow.parquet as pq
 
table = pa.table({
    "timestamp": [0.0, 0.033, 0.066],
    "episode_id": ["ep_014", "ep_014", "ep_014"],
    "action": [[0.1, -0.2, 0.0, 0.0, 0.0, 0.0, 1.0]] * 3,
    "success": [True, True, True],
})
pq.write_table(table, "ep_014.parquet", compression="zstd")
 
# Read only two columns, filtered — no full-file scan
result = pq.read_table("ep_014.parquet", columns=["timestamp", "action"])

Parquet was built with cloud object storage in mind from early on: the footer holds byte offsets for every column chunk, so a reader on S3 or GCS issues targeted range GETs instead of downloading the file. DuckDB, Spark, pandas, and polars all read it natively, which matters when a data scientist wants to filter a million-episode index by success rate and task label without writing a custom loader.

What Parquet does not do natively is dense N-D tensors. A [480, 640, 3] image doesn't have a first-class column type — it gets flattened into a list column or, more commonly, kept out of the parquet file entirely and referenced by path or offset into a separate video or array file, which is exactly the pattern LeRobot and most modern robot dataset formats use.

The hybrid most pipelines actually run

In practice, almost no serious robot data pipeline picks one format for everything. The dominant pattern, and the one MCAP, LeRobot, and most current dataset tooling converge on, is:

• Parquet for tabular state, action, and episode metadata — joint positions, gripper commands, timestamps, success labels, task strings. This is exactly the columnar-analytics access pattern, and it's what you filter and aggregate over when auditing a dataset.
• Separate encoded video files (typically MP4/H.264 or H.265) for camera streams, referenced by episode offset rather than embedded as array data — because video codecs already solve temporal compression far better than a generic array-compression filter does.
• HDF5 reserved for cases where dense arrays genuinely dominate — point cloud sequences, tactile sensor grids, or any tensor workload where an existing tool expects an HDF5 dataset and rewriting that tool isn't worth it.

This is a division of labor, not a compromise: each format is doing the part of the job its data model actually fits, rather than being stretched to cover the whole pipeline. See choosing a robot data format for how this plays out against MCAP and rosbag for the streaming/logging layer specifically.

The recommendation

Default to Parquet for anything tabular — state, actions, metadata, indices — because the ecosystem around it (DuckDB, Spark, pandas, cloud-native reads) is larger and better maintained than HDF5's for that access pattern, and because it's what the current generation of dataset formats already standardize on. Reach for HDF5 only when you have dense multi-dimensional arrays that a specific tool expects in that shape, or when you're maintaining an existing HDF5-based pipeline where a rewrite isn't justified. If you're starting a new collection pipeline in 2026, keep video out of both formats entirely — encode it separately and reference it by offset, the way LeRobot vs RLDS shows both major dataset formats now do.