A pilot batch of 20 episodes and a production run of 10,000 are different engineering problems, not the same problem run longer. The pilot batch tells you the task and the rig work. It tells you almost nothing about whether your storage pipeline, your QA process, or your operators will hold up two orders of magnitude further out — those failure modes only appear at volume, and by the time they do, they're expensive to fix retroactively. For context on what "at scale" looks like in the field: Open X-Embodiment, the largest public cross-institution effort to date, pools upward of a million episodes across 22 robot embodiments from 60 separate datasets — a scale no single lab reaches by running one rig longer, only by solving the problems below. This guide walks through what breaks, in roughly the order it breaks, and how to gate a scale-up so you find out in a controlled way instead of during a deadline.
Step 1: Know what breaks between 100 and 10,000 episodes
Five things degrade quietly as volume increases, none of which show up in a pilot:
- Storage and upload bandwidth stop being an afterthought once multiple rigs are producing multi-camera video continuously.
- QA turns into the bottleneck the moment reviewing every episode takes longer than collecting them.
- Operator drift — two operators who started out consistent slowly diverge as each develops their own habits in isolation.
- Hardware wear and recalibration needs get worse with cumulative hours, not elapsed calendar time, so they sneak up on a program that's tracking dates instead of usage.
- Metadata rot — fields like protocol version and operator id get filled in reliably by one careful person at 50 episodes a day and start silently going missing once three shifts and two rigs are running in parallel.
Step 2: Fix storage and upload before they wall you off
Multi-camera capture is dominated by video, and video is what fills disks first. As an illustrative order of magnitude, a rig running two to three cameras at 30 fps plus proprioception and force data produces on the order of tens of gigabytes of raw footage per collection day before compression — the exact figure depends heavily on resolution, camera count, and codec, so measure your own rig rather than trusting a generic number.
The fix is architectural, not just "buy more disks": upload and compress continuously during collection instead of batching it at the end of a shift. A rig that stores a full day locally before uploading is one bad drive away from losing a day of expensive human time, and a nightly batch upload job that falls behind compounds every day it's late. Record to an indexed, chunked format like MCAP so a partial or interrupted transfer is still recoverable rather than corrupt.
Step 3: Stop QA from becoming the bottleneck
Reviewing every episode works fine at 50 a day and stops working somewhere well before 500. The fix isn't hiring more reviewers linearly with volume — it's switching review strategy. Sample a stratified subset: every episode from a brand-new operator's first sessions, every episode after a protocol change, and a random percentage of steady-state production from experienced operators. Dataset quality assurance covers sampling design in more depth; the point here is that 100% review is a pilot-stage tool, and holding onto it past the pilot is one of the most common reasons a scale-up stalls.
Step 4: Watch for operator drift
Two operators trained on the same protocol in the same week will still diverge over a month of independent collection — one starts approaching grasps slightly faster, another starts resetting on ambiguous failures instead of recording them. Neither notices, because each is only comparing themselves to their own last session. Catch this with periodic calibration sessions where operators run the same episodes side by side and a reviewer checks the outputs against each other, not just against the protocol document. Hiring and training teleoperators covers how to structure these checks on an ongoing cadence.
Step 5: Plan for hardware wear and recalibration
Grippers lose calibration, camera mounts loosen, cables fatigue at flex points — all faster than a pilot-stage rig running a few hours a week would suggest. Tie recalibration checks to cumulative usage (every N episodes or N rig-hours) rather than a calendar schedule, since a rig running three shifts a day accumulates wear far faster than the same rig running occasional pilot sessions. A camera that's drifted out of calibration doesn't fail loudly — it just produces episodes with a subtly wrong extrinsic that nobody catches until a downstream policy trained on it behaves strangely in exactly the region the camera was misaligned toward.
Step 6: Choose parallel rigs vs. longer shifts deliberately
Both scale throughput, but they scale different costs:
- No new calibration or maintenance surface to manage.
- Cheaper to try first — no capital cost, just scheduling.
- Utilization is easy to measure against a rig you already trust.
- Operator fatigue degrades data quality past a few hours per session.
- One rig is a single point of failure for the whole program's throughput.
- Doesn't add scene diversity the way a physically distinct rig does.
A reasonable default: raise shift length and rig utilization first, since it's the cheaper lever, then add rigs once existing ones are running near capacity and the marginal value of scene diversity from a new physical setup outweighs the added calibration and maintenance load.
Throughput math (illustrative only)
Before comparing rigs and shifts, it helps to have a rough model of what one rig-hour produces — but treat every number here as illustrative, not a benchmark, since actual throughput swings by an order of magnitude with task complexity and interface choice.
Two levers move that number more than anything else: reset time between episodes (a scene that resets itself, or resets in one motion, beats one an operator has to hand-arrange) and episode length relative to task complexity (a 90-second bimanual insertion is not going to hit the same rate as a 10-second single-arm pick). Multiply your own measured rig-hour rate by planned rig-hours per week to get a weekly episode budget, and revisit the estimate after the calibration batch in Step 7 rather than trusting a number from this guide or any other source — your rig, task, and operators are the only ground truth that matters.
Step 7: Gate scaling in stages
Treat a scale-up as a sequence of checkpoints, not an open-ended ramp. A workable staging pattern: pilot (~20 episodes, 100% reviewed) → calibration batch (~200 episodes, 100% reviewed, second operator added) → limited production (~2,000 episodes, sampled QA, storage pipeline load-tested) → full production (parallel rigs, staged operator onboarding, sampled QA with automated alerting on acceptance-rate drops). Each gate has its own pass condition — acceptance rate above a threshold, upload pipeline keeping pace with capture, no unresolved operator-drift flags — and you don't open the next gate until the current one clears. This is slower than uncapped scaling in the first week and faster than it in the first quarter, because every failure mode above gets caught at the scale where it's cheap to fix rather than the scale where it's a dataset-wide cleanup.