A human arm has seven meaningful degrees of freedom and a hand with roughly twenty more; a robot arm might have six, seven, or a different link geometry entirely, and a robot gripper might have one actuated axis instead of five fingers. Motion retargeting is the layer that sits between "what the operator did" and "what the robot is commanded to do," and it exists because human and robot kinematics essentially never correspond one-to-one. Every teleoperation system that isn't a kinematically identical leader-follower pair depends on it, whether the input is a VR controller, a bare-hand camera skeleton, or an exoskeleton with a different link geometry than the follower arm.
Configuration-space versus task-space retargeting
There are two fundamentally different places to do the mapping:
- Configuration-space (joint-space) retargeting maps human joint angles directly onto robot joint angles. It only works when the two kinematic chains are structurally similar — the classic case is a leader arm built with the same DH parameters and joint order as the follower, where each leader joint drives the corresponding follower joint directly. This is why ALOHA-style leader-follower rigs feel so responsive: there is effectively no retargeting error to speak of, because the mapping is close to identity.
- Task-space retargeting instead matches a task-relevant quantity — typically end-effector position and orientation, or fingertip position for a hand — between human and robot, and lets an inverse kinematics solver find whatever robot joint configuration achieves it. This is the only option when the kinematic chains genuinely differ, which covers most VR-controller and vision-based hand-tracking setups.
Most production systems use task-space retargeting for the arm (matching a wrist or controller pose) and a specialized variant for the hand, because a human hand's coupling and joint count rarely map cleanly to a robot gripper or dexterous hand at all.
IK-based end-effector retargeting
For an arm, the standard pipeline tracks a target pose for the operator's wrist or controller, then solves inverse kinematics against the robot's Jacobian to find joint angles or joint velocities that reach it. Near a singular configuration — where the Jacobian loses rank and some end-effector motion becomes unreachable by any combination of joint velocities — a plain Jacobian-inverse solver produces enormous, unsafe joint velocities for a small commanded motion. The standard fix is a damped least-squares (DLS) solve, proposed independently by Nakamura and Hanafusa and by Wampler in 1986, which trades a small amount of tracking accuracy for a Jacobian that stays well-conditioned near a singularity instead of blowing up.
Hand and finger retargeting for dexterous hands
Retargeting a full dexterous hand is harder than retargeting an arm, because human and robot hands rarely share joint count, coupling, or actuation. The dominant approach, used by systems like DexPilot, is fingertip-position retargeting: track the 3D position of each human fingertip (from a depth camera, glove, or marker set) and solve an optimization that finds robot hand joint angles whose own fingertip positions match, subject to the robot hand's joint limits and coupling constraints. This sidesteps the joint-correspondence problem entirely, since it only requires the robot hand to be able to reach similar fingertip configurations, not to have the same internal structure as a human hand.
Scaling, workspace limits, and drift
A human operator's comfortable motion range rarely matches a robot's reachable workspace — the robot's arm may be shorter, its joint limits tighter, or its base mounted differently relative to the task. Retargeting pipelines handle this with:
- Scaling, where operator motion is stretched or compressed (often non-uniformly per axis) to map a comfortable human range onto the robot's usable workspace.
- Clutching, where the operator can disengage, reposition their hand or controller, and re-engage without moving the robot — the teleoperation equivalent of picking up a mouse and repositioning it.
- Workspace-limit clamping, where commanded poses outside the robot's reachable or safe volume are clamped or rejected rather than sent through IK, which can otherwise produce large, unpredictable joint motion right at the boundary.
Each of these adds a step where operator intent and robot motion can silently diverge if not tuned for the specific task and rig. Scaling in particular is a trade-off rather than a free parameter: a large scale factor lets a small hand motion sweep the whole robot workspace, which is efficient for coarse repositioning but makes fine manipulation harder to control precisely, since the same hand tremor that was negligible at 1:1 scale becomes a visible robot motion at 3:1.
Retargeting error as a data-quality problem
For teleoperation used to collect demonstration data, retargeting sits directly in the path between operator intent and the action recorded in the dataset. Lag between hand tracking and rendered robot pose, jitter from noisy hand-pose estimation, or a systematic scale error all get recorded as if they were the demonstrated action — a policy trained by behavior cloning has no way to distinguish "the operator meant this" from "the retargeting pipeline distorted this." Teams that take retargeting seriously as a data-quality lever measure its residual error directly — comparing commanded end-effector pose against tracked operator pose over a validation trajectory — rather than trusting that the pipeline "looks right" during a live session, the same way time synchronization residuals get measured rather than assumed.