Teleoperation puts a human inside a robot's control loop from somewhere else. The operator sees what the robot sees, moves an input device, and the robot moves in response — continuously, without the human deciding a sequence of discrete commands in advance. That continuity is what separates teleoperation from remote control, and it is why latency, not bandwidth, is the number that governs whether a teleoperation system is usable.
The term covers a wide range of systems, from a surgeon driving a laparoscopic manipulator a metre away to an operator on another continent collecting manipulation demonstrations for imitation learning. What they share is the closed loop: sense, transport, display, command, transport, act — repeated fast enough that the human never notices they are doing it.
The loop, stage by stage
Every teleoperation system is the same loop running tens of times per second:
- Sense. Cameras, joint encoders, and often force/torque sensors sample the robot's state.
- Transport. That state is encoded and shipped to the operator over a network.
- Display. The operator's station renders video and, in bilateral systems, reflects forces back through the input device.
- Command. The operator moves — a leader arm, a VR controller, a space mouse, a haptic device.
- Transport back. The command is serialized and sent to the robot.
- Act. The robot's controller converts the command into joint targets and executes, subject to safety limits.
The full round trip is what the human perceives. This is why teams measure glass-to-glass latency — photons hitting the camera sensor to photons leaving the operator's display, plus the return path — rather than network round-trip time, which typically accounts for only a third to a half of the total.
Why the loop time matters
Human motor control is predictive but corrective. When feedback arrives quickly, an operator makes continuous micro-adjustments during a motion. When it arrives late, they cannot attribute what they see to what they just did, so they fall back on a move-and-wait strategy: issue a small motion, stop, wait for the display to catch up, evaluate, repeat. Throughput drops and, in contact tasks, forces spike because corrections arrive after the contact event.
For data collection this matters twice over. A laggy session is slower — fewer episodes per operator-hour — and it also produces worse demonstrations. Policies trained by behavior cloning reproduce the demonstrator's hesitations and overshoots, so the operator's degraded control strategy is baked into the model.
Interfaces: how the human actually drives
The input device shapes the data as much as the robot does.
- Leader–follower (bilateral) arms. A kinematically similar leader arm is moved by hand; the follower mirrors it. Joint-space correspondence is direct, which makes for clean, low-cognitive-load demonstrations. This is the ALOHA-style approach, and it is the standard for bimanual manipulation.
- VR controllers and hand tracking. The operator's hand pose is retargeted to an end-effector pose. Flexible, cheap, and portable across robot morphologies, but retargeting introduces its own error and the operator gets no force feedback unless it is synthesized.
- Space mice and gamepads. Rate-control devices: the input maps to velocity, not position. Good for coarse positioning and inspection, poor for fine manipulation.
- Haptic devices. Force-reflecting input devices close the force loop back to the operator's hand, which materially improves insertion and deformable-object tasks — at the cost of a much tighter latency requirement, since force feedback destabilizes faster than video lag annoys. See bilateral teleoperation for why that loop is the hardest one to keep stable.
Whichever device you pick, the choice propagates into the dataset. A leader arm records joint-space targets; a VR controller records end-effector poses that were retargeted from a human hand. Those are different action spaces, and a policy trained on one will not consume the other without conversion — which is why the interface decision belongs at the start of a program, not after the first thousand episodes.
Teleoperation as a data-collection instrument
Treated as a lab instrument rather than a remote control, teleoperation produces training data with properties that scripted collection cannot match: contact-rich behavior, natural recovery from mistakes, and the long tail of variation that comes from a human adapting to a scene they were not expecting.
What makes that data usable downstream is discipline in the recording layer — every camera stream, joint state, gripper command, and operator input carried on one clock, with episode boundaries and success labels attached at capture time rather than reconstructed later. That is the difference between a pile of logs and a dataset: see how to collect robot demonstration data for the full procedure, and data collection for how the recording layer is built.
Where it fits alongside autonomy
Teleoperation is not the opposite of autonomy; it is the scaffolding around it. Fleets in production usually run a spectrum:
- Full teleoperation for tasks no policy handles yet, and for bootstrapping the first dataset.
- Shared autonomy, where a policy executes and a human corrects. Those corrections are outsized training signal for their volume, since they land precisely on the cases the policy hasn't learned yet rather than being spread evenly across easy and hard moments alike.
- Supervised autonomy, where one operator monitors many robots and intervenes on exception — the operating point that makes fleet economics work.
The trajectory of a well-run program is to move tasks leftward to rightward on that spectrum, using each stage's data to earn the next.