RLDS

RLDS — Reinforcement Learning Datasets — is a data specification, storage format, and toolchain from Google Research for sequential decision-making data. Where a format like MCAP is agnostic about what "episode" even means, RLDS makes the episode the organizing unit: a dataset is a collection of episodes, and each episode is an ordered sequence of steps. That structural commitment is what let RLDS become the shared substrate for Open X-Embodiment, the largest public aggregation of robot manipulation demonstrations.

The episode-of-steps structure

An RLDS dataset is, at the TensorFlow level, a tf.data.Dataset of episodes. Each episode is a record containing:

• steps — a nested tf.data.Dataset, itself a sequence of step dictionaries.
• Episode metadata — user-defined fields such as episode_id and agent_id that describe the whole trajectory rather than any single step.

Each step dictionary carries a fixed set of possible fields, and every step in a given dataset must expose the same fields even if some are optional:

• is_first and is_last — mandatory boundary flags marking the start and end of an episode.
• observation — the state at this step: images, proprioception, or both, depending on the dataset's feature spec.
• action — the action taken from this observation.
• reward — the scalar return for taking that action, if the dataset defines one.
• discount — the discount factor associated with the reward, for value-based RL use.
• is_terminal — whether the episode ended here because the underlying MDP terminated (as opposed to is_last, which just marks the final recorded step).

This is a lossless, research-grade schema by design: it preserves the full temporal structure of an episode rather than flattening it into independent (state, action) pairs, which matters for anything that needs the sequence — behavior cloning with history, sequence models, or actual RL.

A robot manipulation dataset built on RLDS typically leaves reward and discount unset, since imitation learning has no reward signal to log, and relies on observation and action alone plus the boundary flags to delimit demonstrations. The schema tolerates that: fields are optional per dataset, not per step, so a builder simply omits reward and discount from its feature spec entirely rather than filling them with placeholder zeros.

Container

TFRecord, sharded, read via TFDS

Unit of storage

Episode → nested Dataset of Steps

Mandatory step fields

is_first, is_last

Optional step fields

observation, action, reward, discount, is_terminal

Why Open X-Embodiment standardized on it

Open X-Embodiment pulled together demonstrations from more than twenty robot platforms across more than twenty institutions, each with its own native logging format, action space, and sensor suite. Making that mixture trainable with one dataloader required a common episode schema that didn't force every contributor into an identical action representation — RLDS's step dictionary tolerates per-dataset feature specs as long as they're internally consistent, and each lab wrote a DatasetBuilder that converted its native format into RLDS-on-TFDS. RT-X and Octo, among others, train directly against that unified representation.

What RLDS is good at

Because it inherits tf.data's machinery, RLDS datasets stream well: sharded TFRecord files support parallel reads, prefetching, and shuffling across a mixture of datasets without materializing everything in memory. That's the property large-scale JAX and TensorFlow RL pipelines actually need — a distributed actor pool pulling episodes from a multi-terabyte mixture without every worker needing local disk space for the whole thing. For teams already committed to TF/JAX for training, RLDS is close to a solved problem: mixing weights, filtering, and feature transforms are all standard tf.data operations, and loading an Open X-Embodiment-style mixture is a matter of pointing TFDS at the right builders:

import tensorflow_datasets as tfds
 
builder = tfds.builder_from_directory("gs://bucket/fractal20220817_data/0.1.0")
ds = builder.as_dataset(split="train")
 
for episode in ds.take(1):
    for step in episode["steps"]:
        obs = step["observation"]
        act = step["action"]
        first, last = step["is_first"], step["is_last"]

Sharding also makes mixing datasets of wildly different sizes tractable — a training run can assign sampling weights per dataset so a 10,000-episode contribution doesn't get drowned out by a 1-million-episode one, entirely through tf.data.Dataset.sample_from_datasets.

Where the friction shows up

The same properties that make RLDS strong for TF/JAX pipelines make it awkward everywhere else:

• TensorFlow dependency. Reading RLDS means depending on tensorflow-datasets, a heavy dependency to pull into a pipeline that is otherwise pure PyTorch.
• PyTorch ergonomics. There's no PyTorch-native RLDSDataset equivalent to torch.utils.data.Dataset. The common pattern is iterating the TFDS pipeline and converting each batch to tensors at the boundary, which works but adds a translation layer and a second dependency graph to maintain.
• Rewriting cost. TFRecord's internal layout is TF-specific enough that converting an RLDS dataset to another format — or the reverse, converting your own capture into RLDS — is a real engineering task, not a metadata edit. Teams that only ever stored their raw data as RLDS have had to run bulk conversion passes when the ecosystem's center of gravity shifted toward LeRobot and PyTorch-native tooling.

None of this makes RLDS the wrong choice for TF/JAX-centric research; it makes it a format worth generating on export from a lossless raw capture (like MCAP) rather than treating as your only copy of the data, the same caution that applies to any training-framework-specific format. See LeRobot vs RLDS for a head-to-head on when each one is the better default.