Robot demonstration data is what a policy actually learns from: not a video of a task, but a time-aligned record of what the robot observed and what a human (or a scripted controller) did about it, repeated across enough variation that the mapping generalizes past the exact scene it was captured in. The phrase gets used loosely for "any robot recording," which is part of why so much collected data turns out to be unusable for training.
What a demonstration actually contains
A minimally useful demonstration has three synchronized elements: what the robot saw, what state it was in, and what it did next.
- Camera streams
- One or more RGB (sometimes depth) views, time-aligned to the other streams
- Proprioception
- Joint positions/velocities or end-effector pose, sampled at controller rate
- Action / command
- What was actually sent to the controller — not the intended trajectory
- Gripper state
- Open/closed, force, or continuous width, depending on the end effector
- Force-torque (optional)
- Wrist or joint-level F/T for contact-rich tasks
- Language instruction (optional)
- A short task description, needed for language-conditioned policies
- Success label
- Pass/fail, and ideally a failure mode, attached per episode
The action stream is the part people get wrong most often. It has to be what was sent to the robot, not what the operator intended or what a higher-level planner requested. If a safety layer clips a velocity or a low-level controller smooths a step input, the recorded action needs to reflect that — otherwise the policy learns to imitate a command that never actually produced the observed next state, and behavior cloning quietly trains on a lie.
Logs vs. training-ready data
Every robot generates logs by default — teleoperation sessions, autonomous runs, test rigs. Very little of that is training-ready without work.
A log becomes a demonstration once it clears three bars:
- Synchronization. Every stream shares a clock, so the camera frame paired with a joint state is the one that was actually visible when that state occurred, not one arriving 30–80 ms later on an independent capture pipeline.
- Segmentation. The continuous session is cut into episodes with clean start and end boundaries, ideally marked live by the operator rather than reconstructed from heuristics after the fact.
- Labeling. Each episode carries the metadata a training or filtering pipeline needs — task identifier, success/failure, and enough scene description to stratify a dataset later.
Skip any of these and the data still "exists," but every downstream consumer — a training script, a QA reviewer, another team — re-derives the missing structure by hand, which is the slow and error-prone version of work that's nearly free to do once, at capture time.
The quality axes that actually predict policy performance
Volume is the easiest thing to report and the least predictive of what happens after deployment. Four axes matter more:
- Diversity. Variation in object pose, lighting, background, and embodiment detail (which arm, which gripper, which mount) that mirrors what the robot will see in production. A dataset with wide diversity and modest size regularly outperforms a larger one collected in a single scene.
- Coverage of the failure tail. Demonstrations of recovering from a missed grasp or a dropped object teach the policy what happens after a mistake — a large fraction of real-world task time. A dataset with zero recovery examples produces a policy that has never seen what to do once something goes wrong.
- Action-space consistency. If operators switch between position and velocity control, or the control rate drifts across sessions, the same visual scene maps to inconsistent actions across episodes, and the policy learns a smeared, less confident distribution.
- Label fidelity. A success label that's wrong, or a task tag that's inconsistently applied, corrupts any filtering or curriculum built on top of it later — and unlike a missing camera frame, a wrong label doesn't announce itself.
Where this fits in a collection pipeline
Demonstration quality is mostly determined upstream of the dataset — by the recording rig, the operator's skill, and the discipline of the capture pipeline — not fixed afterward by cleaning scripts. A data flywheel that retrains on production interventions inherits this same requirement: every loop iteration is only as good as the demonstrations feeding it, which is why teams that treat data collection as an afterthought tend to plateau regardless of how much compute they throw at training.
Who actually generates the data
Demonstrations come from three sources, and they are not interchangeable. Teleoperation — a human driving the robot in real time — produces the highest-fidelity data for contact-rich or novel tasks, because the demonstrator adapts to the scene as they encounter it rather than following a script. Kinesthetic teaching, where a human physically guides the arm through the motion, works well for simple pick-and-place on collaborative arms but doesn't scale to bimanual or high-speed tasks. Scripted or autonomous collection is cheapest per episode once a task is already partly solved, but it can't bootstrap a task the robot cannot yet do at all, and its "failures" tend to be narrower and less representative than the mistakes a human demonstrator naturally makes and recovers from.
Most production datasets blend the three: teleoperation to seed a task and cover its long tail, scripted or policy-driven collection to add volume cheaply once a baseline works, and — once a policy is deployed — captured interventions that feed the flywheel described above.
A minimal capture checklist
Before a session counts as producing training-ready data rather than a log to be cleaned up later, it's worth confirming, per episode:
- Clock
- Every stream stamped from one source, not per-device system time
- Boundary
- Start/end marked live by the operator, not inferred afterward
- Outcome
- Success/failure recorded before the next episode begins
- Calibration
- Camera intrinsics/extrinsics captured once per rig setup, not assumed
Missing any one of these doesn't make the recording useless, but it does move work from the capture session — where it's cheap — to a data engineer reconstructing it later, where it isn't.