Imitation learning trains a policy by showing it examples of the task done correctly rather than by defining a reward and letting an agent search for behavior that maximizes it. A demonstration is a sequence of observation-action pairs — camera frames and joint states paired with the commands a demonstrator issued — and the policy learns a function from the former to the latter. No reward function has to be designed, no simulator has to model the task's dynamics, and the policy can be trained on data collected directly on the real robot. That combination is why imitation learning, not reinforcement learning, is the default starting point for most manipulation work today.
Demonstrations instead of a reward signal
Reinforcement learning (RL) needs an agent to try actions, observe outcomes, and improve against a numeric reward. On physical hardware that means real exploration: a partially trained policy swinging an arm into failure modes a reward function has to anticipate and penalize correctly, over the thousands of trials most RL algorithms need to converge. Simulation sidesteps the safety problem but introduces the sim-to-real gap instead.
Imitation learning avoids both. A human already knows how to do the task; the learning problem is reduced to reproducing what they did. This makes it dramatically more sample-efficient for a given amount of robot time and lets teams train directly against the sensors, actuators, and dynamics the deployed policy will actually run on. The cost is that the policy is fundamentally capped by the demonstrations — it has no mechanism for discovering behavior better than what it was shown.
Three ways to learn from demonstrations
- Behavior cloning (BC). The dominant approach: supervised learning of an observation-to-action mapping, trained the same way any regression or classification model is trained. See behavior cloning for the mechanics and its specific failure modes.
- Inverse reinforcement learning (IRL). Instead of copying actions directly, IRL infers the reward function that best explains why the expert acted as they did, then runs RL against that recovered reward. In principle this generalizes better to states the demonstrations never covered, because the policy is optimizing intent rather than mimicking motion — but it requires an RL loop on top of the inference step, which makes it far more compute- and engineering-heavy than BC, and it sees comparatively little production use in manipulation.
- Interactive imitation (DAgger-style). Rather than training once on a fixed offline dataset, the expert supervises the learner's own rollouts as training proceeds: the partially trained policy acts, and the expert labels the correct action for the states it actually visits. This directly targets the gap between training and deployment distributions instead of hoping offline data happens to cover it.
Why compounding error is the central failure mode
Standard supervised learning assumes training and test data come from the same distribution. Behavior cloning violates that assumption the moment it is deployed: the policy's own small errors shift it into states slightly outside the demonstrated distribution, where it has no training signal, so it errs a little more, drifting further still. Ross and Bagnell's analysis (Efficient Reductions for Imitation Learning, AISTATS 2010; extended in A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, AISTATS 2011, with Geoffrey Gordon) showed that naively trained behavior cloning accumulates error that grows quadratically with task horizon, while interactive approaches like DAgger — which label the learner's own induced states — bound it linearly instead.
Diversity and recovery beat volume
Because the failure mode is drift away from the training distribution, the fix is not more data of the same kind — it is data that covers the states a slightly imperfect policy will actually reach. Ten thousand identical, perfectly executed episodes of a task teach a policy nothing about what to do the first time its gripper is 2cm off-nominal, because that state was never in the training set. What reduces compounding error is:
- Varying starting poses, object placement, lighting, and backgrounds deliberately across episodes rather than holding a scene fixed.
- Capturing and keeping recovery segments — the moments after a near-miss where a demonstrator corrects course — instead of discarding "imperfect" takes.
- Collecting from multiple demonstrators, since a single operator's habitual path through state space is itself a narrow distribution.
What this means for collection practice
The implication for a data collection program is direct: optimize for coverage of the states near the task's failure modes, not for episode count. In practice that means running collection sessions with teleoperation so a human can improvise recoveries in real time, deliberately perturbing the scene between episodes, and labeling where a demonstration recovered from an off-nominal state rather than only labeling final success or failure. Where an expert can be kept in the loop, periodically running the partially trained policy and having a human correct its own rollouts — the DAgger pattern — closes the gap that pure offline collection cannot, at the cost of needing expert time synchronized with training rather than banked in advance.