Ask a simulation researcher and a teleoperation lead which data source matters more, and you get two confident, opposite answers — because they are usually looking at different tasks. Simulation and real data are not competing sources of the same thing; they are different instruments that fail in different places. The question worth answering is not "which one," but which parts of your task the two can actually cover.
The short version
| Simulation | Real robot data | |
|---|---|---|
| Marginal episode cost | Near zero once task/asset built (illustrative) | Roughly constant, bounded by operator time (illustrative) |
| Ground-truth labels | Free — exact pose, contact, segmentation | Requires instrumentation or post-hoc labeling |
| Contact dynamics fidelity | Approximate; friction and deformables are hard | Ground truth by definition |
| Sensor realism | Rendered; noise and artifacts must be modeled | Real camera/depth/force noise, no modeling needed |
| Scale-up speed | Parallelizable across thousands of environments | Bounded by rigs and operators |
| Safety of exploration | Failures are free — crash, reset, repeat | Failures cost time, hardware wear, and risk |
| Best-suited task class | Locomotion, navigation, coarse reach/place | Contact-rich manipulation, deformables, final eval |
What simulation is genuinely good at
Simulation earns its place in the pipeline in a few specific ways, not as a blanket substitute for the real world.
Scale. Once a task and asset are built, generating the next thousand episodes costs compute, not operator-hours. This is the property that makes simulation attractive for anything where you need broad coverage — thousands of object poses, lighting conditions, or initial states — rather than a handful of careful demonstrations.
Locomotion and navigation. Legged and wheeled locomotion policies, and navigation stacks that reason mostly over free-space and obstacle avoidance, transfer from simulation to reality relatively well. The physics that matters — ground contact forces at a coarse level, collision avoidance, path planning — is within reach of a good rigid-body simulator, and the failure modes in the real world (a slightly different floor friction, a miscalibrated encoder) are the kind of thing domain randomization was built to cover.
Coarse manipulation and reaching. Getting an end-effector to a target pose without fine contact requirements — reaching, coarse placement, obstacle-avoidant approach — is a task class where sim-trained policies frequently transfer with modest fine-tuning.
Free, perfect labels. Every frame in simulation comes with exact object pose, segmentation, contact state, and success labels, for free. Producing the equivalent labels from real sensor data requires either instrumentation you have to build or annotation you have to pay for — see language annotation for robot data for what that actually costs on the real side.
Safe exploration of failure. A simulated arm that drives itself into a wall costs a reset. A real arm that does the same costs a service call and, sometimes, a damaged part. For reinforcement learning or any method that needs to sample failure states, simulation removes the safety tax entirely.
Where simulation struggles
The same properties that make simulation cheap are what make it unfaithful in exactly the places contact-rich manipulation lives.
Contact dynamics. Friction, stiction, and the exact force profile of two rigid bodies touching are notoriously hard to simulate accurately, and manipulation tasks — insertion, mating, tool use, grasping under uncertainty — depend on precisely this signal. A policy trained on a simulator's approximate contact model learns to exploit that approximation, which is a liability the moment it meets a real surface.
Deformable objects. Cloth, bags, cables, and food are common in real manipulation tasks and are still an active research problem in simulation fidelity. Most production simulators either skip deformables or approximate them coarsely, which means a policy trained purely on synthetic deformable manipulation is trained against a physics model that does not resemble the object it will actually touch. See contact-rich manipulation data for what real data collection looks like for exactly this task class.
Sensor realism. Rendered RGB, simulated depth noise, and synthesized force-torque signals are approximations of what a real sensor suite produces. Reality gap in perception compounds with reality gap in dynamics — a policy that has to jointly handle both is transferring across two gaps at once, not one.
Cost curves, honestly
- You need broad coverage of poses, lighting, and initial conditions cheaply.
- The task is locomotion, navigation, or coarse reach/place.
- You need exact ground-truth labels without building instrumentation.
- Exploration involves failure states that would be costly or unsafe in reality.
- The task is contact-rich: insertion, mating, tool use, in-hand manipulation.
- Objects are deformable — cloth, cables, bags, food.
- You are evaluating whether a policy actually works, not just pretraining it.
- Building a faithful-enough simulator asset would cost more engineering time than collecting real episodes directly.
The cost comparison people reach for — "simulation is free, real data is expensive" — is true only for the marginal episode. It ignores the fixed cost of building a simulator asset and task that transfers, which for a genuinely novel object or contact task can rival or exceed the cost of just collecting real demonstration data for the same task. Simulation amortizes well when you reuse an asset across thousands of episodes and many training runs; it amortizes poorly when every new SKU or new contact task needs its own calibrated model.
Mixing sim and real: co-training
The practical answer for most contact-rich manipulation programs is not "sim or real" but a mixing strategy. Two patterns dominate:
- Pretrain on sim, fine-tune on real. Use large-scale simulated data to learn a general visuomotor prior, then adapt on a smaller real dataset that corrects what the simulator got wrong. This works best when the sim task is a reasonable approximation and the real fine-tuning set is large enough to shift the policy's contact behavior, not just its visual features.
- Co-train on a mixture. Train on a blend of sim and real episodes in every batch, with the mixing ratio treated as a hyperparameter. Too much sim and the policy inherits the simulator's contact biases; too little and you lose the coverage benefit that motivated using sim at all. There is no universal ratio published that generalizes across simulators and tasks — it has to be tuned per program, and real data should be weighted toward the tasks that will actually ship.
In both patterns, real data plays a role sim cannot substitute for: it is the anchor that keeps the policy's contact and force behavior grounded in reality, and it is the evaluator — the only trustworthy signal for whether a policy that looks good in simulation will work in your lab.
The recommendation
If your task is locomotion, navigation, or coarse pick-and-place, lean on simulation for scale and treat real data as a validation set. If your task is contact-rich — insertion, deformables, anything where force and friction determine success — real data is not optional, and simulation's role shrinks to pretraining and coverage, not final capability. Most working manipulation programs end up using both, deliberately: sim for the volume and safe exploration a real rig cannot give you, real teleoperated data as the non-negotiable anchor for contact fidelity and the only credible measure of whether the policy actually works. Treat any specific cost or scale number you see quoted for either source as illustrative until you have measured it against your own simulator and your own rig.