A demonstration dataset with no QA layer degrades quietly. Nothing crashes — the training loop runs, the loss curve looks fine, and the resulting policy is just worse than it should be, in ways that are hard to trace back to specific episodes. The fix isn't heroic manual review of everything; it's a pipeline that spends automated compute first and human attention second, on the episodes that actually need it.
Automated checks come first
Anything a script can catch, a script should catch, because human review time is the scarcest resource in the pipeline. Run these on every episode before it reaches a person:
- Dropped frames. Compare the expected frame count from the nominal camera rate against what was actually recorded. A camera that silently drops frames under load produces gaps a downstream resampler will paper over — badly.
- Timestamp monotonicity and gaps. Every stream's timestamps should be strictly increasing, and the gap between consecutive samples should stay within a tight band around the expected period. A backward jump usually means a clock reset; a long gap usually means a stalled publisher. Both corrupt any resampling done later, which is the same failure mode covered in more depth in time syncing sensors onto one clock.
- Action-observation misalignment. If the recorded "action" at time t is actually the command that produced the observation at t+1, every training pair in the episode is silently mislabeled. Check this by correlating commanded joint velocity against the derivative of observed joint position — a consistent lag shows up immediately as an unexplained offset.
- Saturated or blurred camera frames. Compute per-frame overexposure (fraction of near-255 pixels) and a blur proxy (Laplacian variance). Flag frames outside a threshold band rather than trying to hand-tune a perfect cutoff — the goal is routing to review, not final judgment.
- Joint-limit clipping. If commanded or observed joint positions sit at a hardware limit for a nontrivial fraction of the episode, either the task is fighting the robot's range of motion or a retargeting bug is clamping values. Both are worth a look.
- Gripper-state inconsistencies. A gripper that reports "closed" with no corresponding drop in aperture width, or a grasp success flag set without a plausible force or width signature, usually means the label is wrong rather than the grasp.
- Episode length outliers. Episodes far shorter or longer than the task's typical duration are disproportionately likely to be aborted attempts, stalls, or an operator leaving the loop running past task completion.
None of these checks need to be perfectly tuned. They only need to route episodes into "clearly fine," "clearly broken," and "needs a human" buckets — false positives cost review time, false negatives cost dataset quality, and the tuning target is minimizing the second at a tolerable cost in the first.
Human review: what to sample and how
Whatever survives automated screening still needs a human sample, because some failure modes — a subtly wrong grasp point, a task performed in a way that satisfies the letter of the instruction but not its intent — don't show up in signal statistics at all.
Sample by stratification, not by flat random draw. Pull a fixed number of episodes per operator, per task, and per session rather than a uniform random slice of the whole dataset. A skilled operator having a bad day, or one operator consistently cutting a step short, is exactly the kind of localized problem a random sample is likely to miss and a stratified one catches within a session or two.
The accept, reject, or repair rubric
Give reviewers three outcomes and a short, concrete definition for each — vague guidance produces inconsistent calls, which shows up as noisy inter-reviewer agreement and, eventually, as unpredictable data quality.
- Accept
The episode matches its instruction, the interaction looks intentional, and no automated flag was a false negative.
- Reject
The core task didn't happen, a required modality was dead for the episode, or the label is unrecoverably wrong.
- Repair
The episode is salvageable with a bounded, documented fix — relabel success to failure, trim a stalled tail, drop one bad stream and keep the rest.
Repair is the category teams under-use. It's tempting to just reject anything imperfect, but a repair pass that relabels a failed grasp as a failure — rather than discarding the episode — turns a QA cost into training signal, covered next.
Failed episodes are training signal, not trash
The instinct to delete anything that didn't succeed throws away some of the most useful data in the set. A dropped object followed by a recovery grasp, a collision followed by a retreat, an over-shoot followed by a correction — these are exactly the contact-rich, off-nominal transitions a policy needs to have seen at training time, because they're exactly what it will encounter at deployment time.
The requirement is labeling, not perfection: mark the episode with what actually happened (attempt-then-recover, partial success, clean failure) rather than either force-fitting it into "success" or deleting it. A dataset that includes labeled failures and recoveries alongside labeled successes gives a policy — and its evaluation — something a success-only dataset structurally cannot: examples of what going wrong looks like and how it gets fixed.
Tracking QA metrics over time
A single aggregate pass rate hides almost everything useful. Track pass rate broken out by:
- Check type, so a spike in, say, gripper-state flags points straight at a hardware or firmware issue rather than a vague "quality dropped" signal.
- Operator, so a skill or fatigue trend shows up in weeks, not months.
- Session and rig, so a miscalibrated camera or a loosened mount surfaces as a step change in one rig's numbers rather than diffuse noise across the fleet.
- GRANULARITY
- Per check, per operator, per rig — never one number
- CADENCE
- Trended per session, reviewed weekly
- ALERTING
- Step changes matter more than absolute level
Trended this way, QA metrics catch problems while they're still cheap to fix — a camera that's drifted out of calibration is a five-minute recalibration if caught this week, and a dataset-wide relabeling exercise if caught in three months. See calibrating a multi-camera rig for what a drift check actually looks like on the hardware side.
Closing the loop back to operators
QA that only produces a filtered dataset is half a system. The other half is feeding what QA found back to the people generating the data. A recurring flag — the same operator consistently triggering joint-limit clipping on the same task, the same rig producing episodes with a specific gripper-state pattern — is either a training gap or an equipment problem, and both are fixable faster at the source than downstream.
Make the failure taxonomy from your rubric visible to operators directly: which checks their episodes are tripping, and how their pass rate compares to the team's. This closes what is otherwise a one-way pipe from operator to dataset into an actual feedback loop, and it's the mechanism that turns a QA process from a filter into a flywheel — each session's review improving the next session's collection.