An episode is one bounded attempt at a task: the robot (or operator) starts from a defined state, acts for some duration, and reaches an end state — success, failure, or timeout. Everything a dataset format needs to index, split, filter, or shuffle a collection of demonstrations happens at the episode level, which is why getting the boundary right matters more than it looks like it should.
Episode, trajectory, step, rollout — the precise distinctions
These four words get used loosely enough in robotics conversation that it's worth pinning down what each one actually means.
| Term | Meaning | |
|---|---|---|
| Step | Unit | One timestamped (observation, action) pair — the smallest granularity |
| Trajectory | Unit | Any ordered path through state space; can be shorter or longer than an episode |
| Episode | Unit | A bounded attempt with a defined start/end and attached metadata — the indexing unit |
| Rollout | Unit | An episode generated by executing a policy (vs. a human), often used during evaluation |
In practice: "trajectory" describes the geometry of the motion, "episode" describes the bookkeeping around it, and "rollout" specifies who produced the episode — a policy under evaluation, rather than a human demonstrator. A single episode can also be decomposed into sub-trajectories (reach, grasp, transport, place) for analysis without changing where the episode boundary itself sits.
Marking boundaries at capture time vs. reconstructing them
Every episode needs a start and end mark, and there are two ways to get one.
Live marking. The operator presses start before beginning the task and stop (with a success/failure tag) at the end. This costs a keypress and produces an unambiguous boundary, because the person doing the task knows exactly when it started and ended.
Post hoc reconstruction. A script infers boundaries from signals like gripper-state transitions, velocity dropping near zero, or gaps in the action stream. This works passably for simple pick-and-place and fails on anything with a natural pause — a multi-stage task, an operator repositioning mid-sequence, a robot waiting on a conveyor. Reconstructed boundaries also can't recover an outcome label; "success" or "failure" isn't inferable from motion alone.
Variable-length episodes
Episodes are rarely the same length. A grasp attempt might run 3 seconds; an assembly sequence might run 45. Two approaches handle the mismatch between variable episode length and a training pipeline that wants fixed-size batches:
- Padding. Every episode is padded to the longest length in the batch (or a fixed cap), with a mask so the loss ignores padded steps. Simple, but wastes compute on padding when episode lengths vary widely.
- Chunking. Long episodes are cut into fixed-size windows, each treated as an independent training example. This is the default for models that consume action chunks — the window length is set by the model's chunk size, not the episode's natural length, so a 45-second episode becomes many overlapping training windows rather than one padded sequence.
Neither approach requires episodes to be a uniform length at capture time; that constraint belongs to the training dataloader, not the recording rig.
Per-episode metadata
Beyond the step sequence itself, an episode typically carries metadata that filtering, stratification, and QA depend on:
- Task identifier — which task this episode is an attempt at.
- Success/failure, and ideally a failure-mode tag when it failed.
- Operator ID — for tracking per-operator quality and skill curves.
- Scene/setup identifier — lighting, object set, background, so a held-out evaluation split can hold out a whole scene rather than randomly sampled frames.
- Seed or configuration, for simulation episodes where reproducing the exact initial condition matters.
How episodes map onto dataset formats
RLDS makes the episode/step relationship explicit in its schema: a dataset is a tf.data.Dataset of episodes, and each episode is itself a tf.data.Dataset of steps, where every step carries an observation, action, reward, discount, and is_first/is_last/is_terminal flags. LeRobot's newer storage layout takes a different physical approach — concatenating episodes into shared Parquet and video shards — but preserves the same logical unit through an episode index that maps each logical episode to its byte and frame ranges within those shards. Both formats agree on what an episode is; they differ only in how it's laid out on disk.
Why the episode is the right unit to split on
Almost every operation a dataset pipeline needs — train/validation splits, filtering by task or operator, deduplication, computing per-task success rates — is naturally expressed at the episode level, not the step level. Splitting at the step level instead is a common and costly mistake: because consecutive steps within an episode are highly correlated (the same grasp, seen a frame apart), a random step-level split leaks near-duplicate frames across train and validation, which inflates validation performance without the policy actually generalizing any better. Splitting on whole episodes — and, for anything intended to test generalization, on whole scenes rather than just episodes — keeps the evaluation honest.
Episode-level metadata is also what makes a dataset queryable for QA and curation. "Show me every failed episode from operator 7 on the insertion task" is a filter over episode records; answering the same question at the step level would mean re-deriving episode identity from raw step sequences first.
Common mistakes with episode boundaries
A few patterns show up repeatedly in datasets that were collected without live boundary marking:
- Bleed between episodes. The last few steps of one attempt and the first few of the next end up inside the same episode record, usually because the boundary heuristic triggered late. Training on this teaches the policy an action that doesn't belong to the task it's conditioned on.
- Missing failure episodes. Teams sometimes discard failed attempts rather than labeling and keeping them, on the assumption that only successes are useful. This throws away exactly the recovery behavior and failure-tail coverage a policy needs to handle the real world, where not every attempt succeeds on the first try.
- Inconsistent episode granularity across sessions. If one operator marks a whole multi-object sorting run as one long episode and another marks each object as a separate episode, per-episode statistics (success rate, duration) stop being comparable across the dataset without extra normalization.