Reproducible Robot Datasets: From Capture to Citation

A reviewer asks a simple question: "can you reproduce Figure 4 with a different lab's rig?" For most robot learning papers, the honest answer is no. Not because the policy is wrong, but because the dataset behind it was never built to travel. The gripper's force-torque calibration lived in a lab notebook, the camera intrinsics were baked into a one-off undistort() call that nobody versioned, and the "cleaned" episodes silently dropped frames that never made it into the paper's appendix.

We build capture infrastructure for teleoperated data collection, so we end up debugging this failure mode constantly — usually months after the original session, when someone tries to retrain on a policy that no longer converges. This post is about the provenance chain that prevents that: what to record at capture time, how to version it, how to document it, and how to publish it so someone outside your lab can actually reuse it.

Why robot datasets rot

Vision datasets degrade gracefully — a mislabeled class is a nuisance. Robot datasets degrade catastrophically, because the observation-action mapping is only valid under the exact sensing and actuation configuration that produced it. Four failure modes account for most irreproducible results we see:

• Undocumented calibration. Camera-to-gripper extrinsics, force-torque sensor bias, and joint encoder offsets drift between sessions. If the calibration used for episode 400 differs from episode 4,000 and neither is recorded, a policy trained across the boundary is learning noise.
• Missing sensor configs. Exposure, gain, and white balance on RGB streams; filter cutoffs on F/T sensors; the depth camera's active vs. passive stereo mode. None of these show up in the action tensor, but all of them shift the input distribution.
• Silent post-processing. Smoothing splines applied to joint trajectories, outlier episodes dropped for "quality," timestamps resampled to a fixed rate — each is defensible, none is reproducible unless the transform and its parameters are recorded alongside the raw capture.
• Format drift. A dataset saved as ad hoc HDF5 in 2023 with a particular key schema is unreadable by anyone who didn't get the original loader script. Open X-Embodiment's own changelog documents multiple schema migrations for exactly this reason.

The fix isn't more discipline from individual grad students — it's making the provenance chain a structural property of the capture pipeline rather than an afterthought.

The provenance chain

Think of provenance as three layers, each with a different failure mode if skipped.

Capture metadata. Every episode needs a header, not just a trajectory. At minimum: rig serial number, calibration hash (a content hash of the calibration file, not just a filename), firmware version for the controller and any embedded microcontrollers, sensor sample rates, and the clock synchronization method. If your rig uses PTP (IEEE 1588) for cross-sensor sync, log the achieved offset, not just "PTP enabled" — sync quality varies with network load and matters for contact-rich tasks where a 5ms skew between F/T and RGB timestamps corrupts the correlation a diffusion policy is supposed to learn.

Content-addressed versioning. Datasets change — episodes get added, a subset gets re-annotated, a corrupted shard gets replaced. Track this the way ML-Ops tracks models: content hashes per shard (SHA-256 over the serialized episode, not the file mtime), a manifest that maps dataset version to shard hashes, and immutable releases. DVC, Git LFS with a hash-addressed remote, or a simple Parquet manifest all work; the mechanism matters less than the invariant — a version tag must always resolve to the same bytes.

Episode-level QA records. Aggregate dataset stats hide the failures that matter. Record pass/fail per episode against explicit criteria: did the gripper close within tolerance, did any sensor stream drop frames, was the episode manually flagged during collection. Downstream users filter on this instead of re-discovering your exclusion criteria by trial and error.

Datasheets for datasets

Gebru et al.'s "Datasheets for Datasets" (2018/2021) proposed a standard set of questions — motivation, composition, collection process, uses, distribution, maintenance — answered once per dataset and shipped alongside it. For robot data we extend the template with hardware-specific fields, because "composition" for a manipulation dataset means sensor modalities, control frequency, and workspace geometry, not just row counts.

A minimal machine-readable manifest looks like this:

dataset:
  name: bimanual-insertion-v3
  doi: 10.5281/zenodo.1234567
  version: 3.1.0
  license: CC-BY-4.0
  episodes: 4820
  hz: 30
rig:
  serial: telemanual-rig-0091
  arm: ur5e
  gripper: robotiq-2f-140
  ft_sensor: ati-axia80
  calibration_hash: sha256:9f2e1a...c04b
  firmware:
    controller: 4.2.1
    gripper_driver: 1.9.0
sync:
  method: ptp-ieee1588
  measured_offset_ms: 0.8
cameras:
  - id: wrist_cam
    model: realsense-d405
    resolution: [640, 480]
    fps: 30
    intrinsics_hash: sha256:1c77bd...ee31
qa:
  episodes_flagged: 63
  exclusion_criteria:
    - "gripper_timeout > 2s"
    - "ft_sensor_dropout > 5 frames"
provenance:
  collected_by: lab-teleop-team
  collection_dates: ["2026-02-01", "2026-04-18"]
  postprocessing:
    - step: "trajectory resample to 30hz"
      tool: "scipy.interpolate.interp1d"
    - step: "outlier episode removal"
      criteria: "manual QA flag"

Publishing so it actually gets reused

Open X-Embodiment and BridgeData set a useful bar: both publish with explicit format documentation, a defined episode schema, and a stable Hugging Face hub presence rather than a one-time tarball drop. That combination — hub hosting plus a documented schema — is what lets a third party load the dataset with a few lines of code instead of reverse-engineering a loader.

For export, standardizing on open formats avoids the loader-script problem entirely:

LeRobot's dataset format (v3 schema) and RLDS/TFDS both define episode structure explicitly enough that a policy training script written against the spec works across datasets that follow it — this is a meaningful part of why Open X-Embodiment's 22-dataset, 21-embodiment aggregation was feasible at all. MCAP is the better choice when you need to preserve full sensor fidelity (multiple camera streams, F/T at native rate, ROS 2 topics) for later re-derivation into a training format, since it's a general-purpose timestamped-message container rather than an ML-training-shaped schema.

Licensing deserves a deliberate choice, not a default. CC BY 4.0 maximizes reuse and citation — most Open X-Embodiment constituent datasets use it — but if the data includes participant likeness or proprietary rig designs, a more restrictive research-only license (CC BY-NC, or a custom academic-use agreement) may be warranted. Whatever you choose, state it in both the datasheet and the hub repo's metadata; ambiguous licensing is one of the most common reasons datasets get quietly excluded from follow-up aggregations like Open X-Embodiment.

Finally, make it citable as an artifact, not just a mention in a methods section. Mint a DOI (Zenodo and Hugging Face's DOI integration both work) and publish a BibTeX entry:

@misc{bimanual_insertion_v3_2026,
  title        = {Bimanual Insertion Dataset v3},
  author       = {Lab Teleop Team},
  year         = {2026},
  doi          = {10.5281/zenodo.1234567},
  url          = {https://huggingface.co/datasets/lab/bimanual-insertion-v3},
  note         = {Version 3.1.0}
}

Checklist: what to record before a reviewer asks

None of this is exotic — it's the same rigor wet-lab biology has applied to reagents and protocols for decades. The gap in robotics is tooling, not willingness: capturing rig serials and calibration hashes at record time is a one-line addition to a session header, but only if the capture pipeline supports it natively. That's the design point we optimized for in Telemanual's capture layer — every teleop session is stamped with rig, calibration, and firmware metadata automatically, exports carry a manifest by default, and a dataset gets a DOI and BibTeX entry from the same interface used to browse episodes. The goal isn't a prettier README; it's a dataset that still means the same thing to someone else's lab a year from now.