Most bad robot datasets aren't bad because the robot was unreliable — they're bad because nobody wrote down what "success" meant before the cameras started rolling. Robot demonstration data is expensive to produce and unforgiving to fix after the fact: a missing timestamp, an unlabeled failure, or an ambiguous task boundary doesn't show up until someone tries to train on it weeks later. This guide walks through the seven steps that keep that from happening, from defining the task through exporting a versioned dataset, with a worked example and a pre-flight checklist you can hand to an operator on day one.
- Define the task and success criterion
Write down the task boundary and the exact, checkable condition that separates success from failure.
- Choose the teleoperation interface
Match the input device to the task's precision and force requirements.
- Instrument the rig
Put every camera, sensor, and the gripper on one shared clock.
- Write the collection protocol
Turn the task definition into a versioned document any operator can follow identically.
- Run a pilot batch and inspect it
Collect ~20 episodes and look at the raw data before scaling.
- QA and label the episodes
Check every episode, tag failures and recoveries, catch sync problems early.
- Export and version the dataset
Convert to a training format and tag a dataset version.
Step 1: Define the task and success criterion
Before anyone touches an input device, write two sentences: what the robot is supposed to do, and how you will know, looking only at the recorded data, whether a given episode achieved it. "Pick up the object and place it in the bin" is not a success criterion — it doesn't say what counts as placed (released and stationary? inside a marked region?), and it says nothing about what to do if the object is dropped and re-grasped.
A usable criterion is checkable by someone who wasn't in the room: "Success if the object's centroid is inside the bin footprint and the gripper is open and retracted, within 15 seconds of episode start." Write down the failure modes you expect too — dropped grasp, missed approach, object out of reach — because you'll reuse this list when you get to QA.
Step 2: Choose the teleoperation interface
The input device is not a peripheral choice — it determines the character of every trajectory you collect. Teleoperation interfaces trade off precision, force feedback, and cost differently:
- Leader-follower arms give the cleanest joint-space correspondence and the lowest cognitive load for the operator, which is why they're the standard for contact-rich bimanual work.
- VR controllers and hand tracking are cheap and portable across robot morphologies, at the cost of retargeting error and no native force feedback.
- Space mice and gamepads work for coarse positioning and inspection tasks but struggle with fine manipulation because they're rate-control, not position-control.
- Haptic devices close the force loop back to the operator's hand, which measurably improves insertion and deformable-object tasks, but need a tighter latency budget than video alone.
If the task involves contact — insertion, stacking, cloth, anything where force matters — bias toward leader-follower or haptic. If it's mostly free-space motion, VR or a space mouse will save setup time. See teleoperation vs. kinesthetic teaching if you're also weighing physically guiding the arm by hand against remote control.
Step 3: Instrument the rig
Four categories of signal need to land in every episode, all referenced to the same clock:
- Cameras. At minimum one scene-view and one wrist-mounted camera; add overhead or side views for tasks with occlusion. 30 Hz RGB is a reasonable default; go to 60 Hz for fast contact events.
- Proprioception. Joint positions and velocities (and torques, if your controller exposes them) at a rate at or above your camera rate.
- Gripper / end-effector state. Commanded and measured gripper width or force, plus end-effector pose if your policy will consume Cartesian actions.
- Clock. One shared time source all of the above are stamped against — this is the single most common thing that's missing from a dataset someone tries to reuse six months later.
If your rig uses more than one camera, run multi-camera calibration before the pilot batch, not after — an uncalibrated camera set is expensive to fix retroactively. For the clock problem specifically, see time-syncing sensors to one clock.
- SCENE CAMERA
- Fixed mount, task fully in frame, 30–60 Hz
- WRIST CAMERA
- Rigidly mounted to end-effector, matched exposure
- PROPRIOCEPTION RATE
- ≥ camera rate, no interpolation needed
- CLOCK
- Single source; every stream timestamped against it
Step 4: Write the collection protocol
The task definition from Step 1 becomes a document, not a shared understanding in people's heads. A protocol specifies scene setup, object set and randomization, episode start and end conditions, and what an operator does on a failure — reset silently, or record the failure and reset? Designing a robot data collection protocol covers this in depth; the short version is that a protocol is only doing its job if two different operators, reading it cold, would produce data indistinguishable in aggregate.
Step 5: Run a pilot batch and inspect it
Collect on the order of 20 episodes before you commit to a production run. This is cheap enough that finding a problem here costs an afternoon, and expensive enough to surface real issues — a single episode won't show you clock drift, but twenty will.
Inspect the raw data, not the dashboard summary: play back video against joint trajectories and confirm they're in sync, spot-check that camera exposure didn't drift between episodes, and confirm the success criterion from Step 1 is actually checkable from what got recorded. If it isn't — say, the bin isn't visible in any camera frame at the end of the episode — you have a rig problem, not a labeling problem, and it's much cheaper to fix now than at episode 2,000.
Step 6: QA and label the episodes
Every episode gets checked against the success criterion, and every failure gets classified rather than discarded — a dropped grasp and a policy-relevant near-miss are different training signal. This is also where you catch sensor and sync problems that slipped past the pilot: a camera that silently dropped frames, a gripper encoder that stuck. Dataset quality assurance covers sampling strategies for QA once volume makes reviewing every episode impractical — which happens sooner than most teams expect.
Step 7: Export and version the dataset
Convert the raw capture into a training-ready format — LeRobot, RLDS, or MCAP-backed storage are the common choices, and choosing a robot data format walks through the tradeoffs. Attach episode-level metadata (task id, success label, operator id, protocol version) at export time rather than reconstructing it later, and tag the resulting dataset with a version. A dataset without a version is a dataset nobody can cite when a downstream policy's behavior changes.
Worked example: connector-mating pilot
Say the task is plugging a barrel connector into a charging port — contact-rich, low tolerance, a good stress test for a rig. The success criterion: connector fully seated (measurable as a step change in insertion-axis force) within 10 seconds, gripper retracted without disturbing the connector. Interface: a leader-follower arm, since the insertion force feedback matters and free-space retargeting error would make fine alignment painful. Instrumentation: wrist camera for the approach, scene camera for context, 6-axis force/torque at the wrist, joint states at 200 Hz, cameras at 30 Hz, all stamped to the controller's clock. Protocol: 15 connector positions randomized within a marked zone, three lighting conditions rotated across sessions, failures recorded and classified as missed-approach, cross-threaded, or dropped. Pilot batch: 20 episodes, reviewed same day — this is typically where you discover the force sensor is sampling slower than you assumed, or that the wrist camera's field of view clips the connector at full insertion. Only after that batch checks out does the protocol go to a second operator for a calibration session, and only after that does the program scale.
Pre-flight checklist
- Success criterion is written down and checkable from recorded data alone, not from memory of "how it looked."
- Every camera, sensor, and the gripper share one clock; verify with a clap test or LED flash before the first real episode.
- The protocol document exists as a file, has a version number, and someone other than its author has read it.
- A failure taxonomy exists — you know what you'll call a dropped grasp versus a missed approach before it happens the first time.
- Storage and export format are decided before collection starts, not after 500 episodes exist in whatever format the recording tool defaulted to.
Getting these five things right before you scale is the difference between a dataset you can hand to a training run in a week and one that needs a month of forensic cleanup first. If you're deciding between running this in-house or handing it to a data collection partner, the process above is what "good" looks like either way — the steps don't change, only who executes them.