Hugging Face Hub

The Hugging Face Hub is where models and datasets get hosted as versioned, git-backed repositories — clone-able, diff-able, and addressable by commit the same way code is. For robot learning it does double duty: it's a generic dataset host, and it's also the intended distribution point for LeRobotDataset, whose Parquet-plus-video layout was designed with Hub hosting in mind from the start. Publishing a robot dataset there and making it directly loadable by any lerobot-train run are, in practice, the same action.

How the data gets there

A dataset repo on the Hub is a git repository under the hood, with large files (Parquet shards, MP4 video, checkpoints) tracked through Git LFS rather than committed directly. Pushing a Telemanual export or a self-recorded LeRobotDataset built with LeRobot is a single call:

dataset.push_to_hub("your-org/pick-place-v1", private=True)

or, for arbitrary files and folders rather than a LeRobotDataset object specifically, the hf CLI (the current name for what used to ship as huggingface-cli):

hf auth login
hf upload your-org/pick-place-v1 ./local_dataset_dir --repo-type=dataset

Every dataset repo carries a dataset card — a README.md with a YAML frontmatter block for license, tags, and task metadata, followed by free-text documentation. For a robot dataset, the card is what makes the Parquet rows and video shards interpretable to someone who wasn't in the room: embodiment, camera placement and intrinsics, control frequency, what a success label means, and the collection protocol that produced it. None of that is recoverable from the data files themselves.

Once pushed, a Parquet-backed dataset gets the Hub's built-in dataset viewer for free — a browsable preview of rows with no download required, generated automatically from the file layout rather than hand-built per dataset. For the LeRobotDataset layout specifically, the community-maintained lerobot-dataset-visualizer goes further, rendering episodes with their camera feeds and plotted state/action curves side by side, which is a faster way to spot-check a dataset that just landed on the Hub than pulling it locally and writing inspection code.

Access control and versioning

Repos are public or private, with private access controlled per-user or per-organization and a gated middle option: the repo is visible and citable, but downloading requires an access request that the author approves — automatically or manually — a common choice for a dataset tied to a paper under review. Organizations add resource groups and, on Enterprise plans, finer controls like blocking access requests from specific countries.

Because every repo is a git repository, versioning is git versioning: commits, branches, and tags, all resolvable through a revision parameter.

from huggingface_hub import snapshot_download
 
# Pin to a specific tagged release rather than "main"
path = snapshot_download(
    repo_id="your-org/pick-place-v1",
    repo_type="dataset",
    revision="v1.0",
)

load_dataset, snapshot_download, and LeRobotDataset all accept the same revision argument, so a paper or a model card can point at an exact, immutable snapshot of a dataset instead of a moving branch.

Storage limits and large files

Robot datasets are video-heavy, so file-size limits matter more here than in most Hub use cases. As documented on the Hub's storage-limits page: a hard per-file cap of 500 GB enforced by LFS, with Hugging Face recommending files stay well under that (its own guidance suggests chunking below roughly 200 GB for practical download and diff behavior), and a public-repository storage allowance that scales by plan — a free account gets best-effort public storage, PRO adds 10 TB of public plus 1 TB of private storage, Team adds 12 TB public plus 1 TB private per seat, and Enterprise scales further, up to roughly 1,000 TB. These are the kind of numbers that shift as the platform evolves — check the current page before sizing a collection campaign around them, and treat LeRobotDataset's built-in shard packing (many episodes per Parquet/MP4 file rather than one file per episode) as the mechanism that keeps a growing dataset inside those limits without manual intervention.

The workflow in practice

Publishing a dataset alongside a paper or a model release tends to follow the same sequence regardless of team size:

1. Finalize the dataset — for a LeRobotDataset, that means calling dataset.finalize() before push_to_hub(), since skipping it leaves the Parquet footer unwritten.
2. Write the dataset card before making the repo public, not after — embodiment, protocol, and license belong in the card from the first commit a reviewer might see, not backfilled once someone asks.
3. Choose public, gated, or private based on where the associated work stands — private during collection, gated while a paper is under review, public and DOI-tagged once it's ready to be cited.
4. Generate a DOI from the repo settings once the dataset is stable, so Cite this dataset produces a citation reviewers and downstream users can rely on — a new DOI can be issued for a later version without invalidating the one already cited in print.
5. Pin the revision used in any published result, in the paper text and in any accompanying code, not just the repo ID.

This is the same reproducibility discipline the reproducible datasets field guide walks through end to end, and it's the workflow Telemanual's research tooling is built to make a one-click export rather than a manual checklist.

Gotchas

• A repo ID alone is not a citation. Without a pinned revision, "we used dataset X" describes a moving target, not a specific set of bytes.
• finalize() is easy to forget in a custom capture pipeline, and the failure is silent at push time — it only surfaces when someone tries to load the dataset later.
• Gating adds friction on purpose. If a dataset needs to be immediately usable by an automated pipeline (a CI job, a scheduled retraining run), gating is the wrong access mode — use a scoped access token against a private repo instead.
• Large video files dominate storage math. Before assuming a dataset needs an Enterprise storage plan, confirm it's actually sharded well below the per-file cap and encoded efficiently — the size problem is sometimes a re-encoding problem, not a quota problem.