Two operators reading the same one-line task description will collect two different datasets. One centers the object before every grasp; the other doesn't. One resets on a near-miss; the other pushes through and calls it a success. Neither is wrong, exactly — they just weren't given enough information to be the same. A collection protocol is the document that removes that ambiguity, and writing a good one is closer to writing a test spec than writing instructions for a person: every branch has to be covered, and nothing gets left to judgment calls made in the moment.
Step 1: Write the task spec operators can't misread
Start from the task definition described in how to collect robot demonstration data, then push it one level more literal. "Pick up the mug and place it on the coaster" leaves out whether the mug can be picked up by the handle or the body, whether "on the coaster" tolerates partial overlap, and what happens if there's no coaster in frame. A protocol resolves every one of those before an operator asks.
Write it as a short numbered list, not prose — numbered steps are what an operator actually scans mid-session, and prose gets skimmed past.
Step 2: Schedule scene randomization
A protocol that fixes the scene produces a policy that memorizes the scene. Define a randomization schedule across four axes:
- Object set. Which specific instances appear, and how often the set rotates — every episode, every N episodes, or every session.
- Position. The region within which objects can be placed, and whether placement is operator-judged "roughly here" or measured against a grid or template.
- Lighting. Whether sessions run under fixed lighting or rotate through a small set of conditions (overhead only, side light, dimmed) across the collection window.
- Distractors. Objects present in the scene that are not part of the task — critical for teaching a policy what to ignore, and easy to forget entirely if the protocol doesn't call it out.
Step 3: Define episode start and end conditions
An episode needs an unambiguous trigger to start and an unambiguous trigger to end, and the end trigger has to fire regardless of whether the episode succeeded. A common failure mode: the protocol only defines what "done" looks like for a success, so failed episodes run indefinitely until the operator gets impatient and resets at an inconsistent point, corrupting the tail of the trajectory.
Specify both explicitly: "Episode starts when the arm reaches the marked home pose and the operator presses record. Episode ends at task success, at operator-declared failure, or at a 30-second timeout — whichever comes first." That third clause, the timeout, is what keeps a stuck episode from silently becoming a 4-minute outlier that skews your duration statistics.
Step 4: Decide what counts as success
Success has to be checkable from what actually gets recorded — video, proprioception, gripper state — not from what the operator remembers happening. If your criterion depends on something no sensor captured (did the object settle, or is it still wobbling?), either add the sensor or change the criterion. Enumerate expected failure modes here too, since you'll reuse this list for QA.
Step 5: Decide how to handle failures and recoveries
This is the decision protocols most often skip, and it matters more than almost anything else in this guide: record failures and recoveries, don't discard them. A dataset of only clean successes teaches a policy what to do when everything goes right and nothing about what to do when a grasp slips or an approach angle is off — which is exactly the situation a deployed policy will eventually face with no human in the loop.
- Captures recovery behavior a pure-success dataset never shows.
- Failure rate becomes a measurable signal for task or rig problems.
- Near-misses are cheap to relabel later if labeling criteria change.
- Simpler to run and analyze if you only care about success episodes.
- Risk of an operator conflating a real failure with an aborted setup.
- Requires an explicit failure taxonomy the protocol has to define.
If you do record failures, the protocol needs a short taxonomy — missed approach, dropped grasp, wrong target — so operators tag consistently rather than everyone inventing their own labels. Crowdsourced vs. expert demonstrations covers how this decision interacts with who's collecting the data.
Step 6: Version the protocol as an artifact
Treat the protocol document the way you'd treat code: it lives in version control, it has a change log, and every episode's metadata references the version it was collected under. This sounds like overhead until the first time someone asks why episodes from March behave differently from episodes from June, and the answer turns out to be a protocol change nobody recorded.
- STORAGE
- Plain text or markdown, in version control
- EPISODE METADATA
- protocol_version field on every episode
- REVIEW
- Second operator reads it before first production run
- CHANGE LOG
- One line per revision: what changed and why
Step 7: Change the protocol without splitting the dataset silently
Protocols change mid-program for good reasons — you discover a scene setup that produces bad data, or a new object needs to enter the rotation. The mistake is changing the instructions without changing the version number: episodes before and after the change look identical in the metadata but were produced under different rules, and nobody downstream can tell.
The fix is mechanical: bump the version number, note the change and the date in the log, and make sure the episode metadata pipeline picks up the new version automatically rather than requiring someone to remember to update it. If the change is significant — a new success criterion, not just an added object — treat episodes on either side of the change as two datasets you can union later, not one dataset you're hoping stays statistically identical. This is also the point where it's worth a short operator calibration session so everyone running the new protocol is interpreting it the same way before volume ramps back up.
Worked example: a bin-picking protocol excerpt
A short excerpt makes the abstract version of this guide concrete. For a bin-picking task with five object classes:
- Task spec. "Pick one instance of the target class, indicated by the operator's on-screen label at episode start, and place it in the marked output tray without touching other objects in the bin."
- Randomization. Object set rotates every 5 episodes across a pool of 12 instances per class; bin fill level alternates between sparse (3–4 items) and cluttered (8–10 items) every session; lighting fixed within a session, rotated across three presets between sessions.
- Episode boundaries. Starts on operator-triggered record after the arm reaches home pose; ends at success, at operator-declared failure, or at a 45-second timeout.
- Success. Target object's centroid inside the tray footprint, gripper open and clear of the tray, no other bin object displaced by more than 5 cm.
- Failure handling. Recorded and tagged as one of: wrong-object grasp, dropped-in-transit, or collision-with-neighbor. Operator resets the bin to the next scheduled configuration rather than restoring the exact prior arrangement.
Notice how little of this is about the robot and how much is about removing decisions from the operator's hands. That's the actual goal of a protocol — not to make the task easier, but to make every operator's version of "easier" the same version.
Keeping it a living document
A protocol that's written once and never revisited drifts out of sync with what operators actually do — someone finds a faster way to reset between episodes, or discovers the lighting instruction doesn't match the rig's actual fixtures, and the informal practice diverges from the written one without anyone updating either. Schedule a short review any time you onboard a new operator, since their questions are the cheapest signal you'll get that the document has a gap, and treat every "actually we've been doing it this way for months" as a protocol bug to fix, not a rule to formalize after the fact.