NVIDIA Isaac Lab

Isaac Lab is NVIDIA's open-source framework for GPU-accelerated robot learning — reinforcement learning and imitation learning environments running on top of Isaac Sim, which itself runs on the Omniverse USD stack. It is where a lot of current locomotion and manipulation research trains policies, and it is increasingly where teams evaluate a policy in simulation before it ever touches a real robot. For a program collecting real demonstration data, Isaac Lab is not a replacement for that data — it is a place to replay it, augment it, and stage a policy before deployment.

The naming history matters if you land on older docs. Isaac Gym was NVIDIA's original standalone GPU physics simulator for RL; it is now deprecated, with legacy downloads still available but no further development. Orbit was a separate robot-learning framework built on Isaac Sim; it was absorbed into Isaac Lab, along with the retired OmniIsaacGymEnvs project. If a tutorial, GitHub issue, or blog post talks about Isaac Gym environments or Orbit configs, treat it as historical — Isaac Lab is the actively maintained successor to both.

How the data gets there

Isaac Lab's own demonstration workflow is native HDF5. record_demos.py captures teleoperated episodes — keyboard, spacemouse, or OpenXR device input — into an HDF5 file per task, and replay_demos.py plays them back inside the simulator to verify the recording before it goes anywhere near training:

# Record teleoperated demonstrations for a task
./isaaclab.sh -p scripts/tools/record_demos.py \
  --task Isaac-Lift-Cube-Franka-IK-Rel-v0 \
  --teleop_device spacemouse \
  --dataset_file ./datasets/lift_cube_demos.hdf5 \
  --num_demos 20
 
# Replay and sanity-check the recording
./isaaclab.sh -p scripts/tools/replay_demos.py \
  --task Isaac-Lift-Cube-Franka-IK-Rel-v0 \
  --dataset_file ./datasets/lift_cube_demos.hdf5

Isaac Lab Mimic then takes a small set of these demonstrations — as few as one per subtask — segments them into object-centric subtasks, and generates a much larger set of synthetic demonstrations by transforming and recombining segments across randomized object poses. It is a data-augmentation step, not a substitute for having captured real behavior in the first place: Mimic amplifies the geometric variation around a demonstrated skill, it does not invent a skill that was never shown.

The other entry point for real data is the asset, not the trajectory. Isaac Lab's UrdfConverter wraps the Omniverse URDF importer to turn a URDF robot description into USD, the format Isaac Sim actually simulates. This is how a real robot cell becomes a digital twin: import the robot's URDF, place USD representations of the workcell and fixtures at their measured real-world poses, and calibrate camera intrinsics/extrinsics to match the physical rig. The conversion is lossy on physical parameters — inertia, contact stiffness, joint damping — so a twin built this way needs a validation pass against real trajectories before anyone trusts what it reports.

The workflow in practice

Teams running a simulation-first evaluation loop alongside real data collection tend to converge on a few recurring patterns:

• Digital twin from a real cell. Import the exact robot URDF and workcell geometry used on the shop floor, so a policy trained partly in sim sees the same kinematic chain and roughly the same scene layout it will face for real.
• Replaying real trajectories in sim. Recorded joint or end-effector trajectories from real teleoperation sessions are stepped through the simulated robot to check for kinematic consistency — did the recorded motion stay inside the real robot's joint limits and workspace, and does the simulated end effector trace the same path.
• Sim-and-real co-training. A policy trains on a mixture of real demonstrations and Mimic-augmented or fully synthetic sim rollouts, using sim to cover pose and lighting variation that would be expensive to re-collect physically, while the real data anchors the policy to actual contact dynamics and sensor noise.
• Pre-deployment gating. Before a checkpoint goes on the real robot, it runs a batch of parallelized sim rollouts — Isaac Lab's GPU-parallel environments make thousands of rollouts cheap — as a coarse filter. A policy that fails badly in sim rarely deserves real-robot time; a policy that passes in sim still needs the real test.

Gotchas

Determinism holds on the same machine, not necessarily across machines. Isaac Lab's own reproducibility documentation is explicit that a fixed seed reproduces results on the same hardware and Isaac Sim version, but does not promise bit-exact replay across different GPUs, drivers, or Isaac Sim releases — GPU work-scheduling can introduce least-significant-bit divergence that compounds over a long rollout. Replaying a demonstration recorded on one workstation against a different GPU or a newer Isaac Sim install is where a small, otherwise-inexplicable replay mismatch is most likely to come from.

URDF-to-USD conversion is lazy and cached. The UrdfConverter only regenerates the USD asset when the URDF itself changes — editing a referenced mesh file without touching the URDF silently leaves you simulating the old geometry. Force regeneration or delete the output directory when mesh assets change.

Version churn. Isaac Lab has moved fast: pip packaging, task naming conventions, and the Mimic API have all changed across recent releases. Pin a version and read the docs for that exact release rather than trusting a search result or an older tutorial's flags verbatim.

GPU and driver requirements. Isaac Sim's rendering and physics both expect an NVIDIA GPU with a specific driver range; this is a heavier local dependency than MuJoCo, which runs meaningful workloads on CPU.

PHYSICS ENGINE

PhysX (via Isaac Sim)

SCENE FORMAT

USD (Universal Scene Description)

PARALLEL ENVS

GPU-batched, thousands per run

DEMO AUGMENTATION

Isaac Lab Mimic (subtask segmentation + recombination)

Where real data still wins

Isaac Lab is strongest as a staging ground: build the twin, replay what you already captured to check it against the model, augment a small demonstration set geometrically, and filter policies before spending real-robot time on them. It is weakest exactly where the contact-rich manipulation research it is used for actually lives — friction, compliance, and deformable contact are the hardest things to simulate faithfully, which is also why real, physically-collected demonstrations for those tasks stay valuable regardless of how much cheaper simulated rollouts get. The two are complementary inputs to the same policy, not competing sources of ground truth.