A vision-language-action model is only as good as the mapping it learns between instructions and actions, and that mapping is defined entirely by the language annotation in the dataset — not by the demonstrations alone. A robot arm picking up a red block is training data for "pick up the block" only if that's what the instruction actually says, and it's training data for the wrong thing if the instruction says "pick up the cup" because an annotator was moving fast.
Why VLAs need instructions at all
Behavior cloning without language conditioning learns a single policy per task. A vision-language-action model conditions on an instruction so one policy can cover many tasks, generalizing to instructions it never saw verbatim during training by learning the compositional structure of the language — "pick up X," "place X on Y," "open the Z." That generalization is only as good as the diversity and accuracy of instructions the model was trained on. A dataset where every episode of a given task carries the identical instruction string teaches the model to key off surface phrasing rather than meaning, and it falls over the first time a user phrases a familiar request slightly differently.
Choosing instruction granularity
Three levels of granularity cover most datasets, and they aren't mutually exclusive — a mature dataset usually carries more than one simultaneously.
- Whole-episode task string. One instruction per episode: "put the mug in the sink." Cheapest to produce, works for single-goal manipulation, and is what most capture-time annotation naturally produces.
- Sub-task segments. The episode is split into labeled phases — "reach for the mug," "lift the mug," "move to the sink," "release" — each with its own start and end time. This is what a policy needs if it's expected to ground language to specific portions of a longer trajectory, not just the episode as a whole.
- Dense per-step captions. A caption at every timestep or short window, describing what's currently happening. The most expensive to produce by far, and usually reserved for a curated subset used for fine-grained grounding research or evaluation rather than the whole dataset.
Match the granularity to what the downstream model actually consumes. Annotating every episode densely when the training recipe only ever uses a whole-episode string is annotation budget spent on a signal nobody reads. A reasonable default for a growing dataset: whole-episode strings on everything, sub-task segments on the subset of tasks that naturally decompose into repeatable phases, and dense captions reserved for a small, deliberately curated slice used to stress-test grounding rather than applied across the board.
Writing a style guide annotators can actually follow
Inconsistent instructions are worse than sparse ones, because a model can't tell the difference between "the task varies" and "the annotator's habits vary." A short, concrete style guide fixes the most common sources of drift:
- Imperative voice, always. "Pick up the red block," not "the robot picks up the red block" or "picking up the block." Consistency here matters more than which voice is chosen.
- Name objects the way a person in the room would. "The red block" beats "object_3" or an internal SKU, and it should match how the object is referred to across the whole dataset, not vary between "mug," "cup," and "coffee mug" for the same physical item.
- No ambiguous pronouns. "Place it on the shelf" is only acceptable if "it" was named in the same instruction; never reference an object established only by the video frame, since the model has to learn the mapping from the words alone at inference time when a user hasn't necessarily seen the scene the annotator saw.
- State the goal, not the motion, unless motion is the point. "Open the drawer" describes intent; "move the gripper forward then down then pull back" describes execution and is rarely what you want the instruction to encode, since the policy — not the instruction — is responsible for figuring out the motion.
Paraphrase augmentation and its risks
Once a verified instruction exists for an episode, generating paraphrases — "pick up the block" / "grab the block" / "get the red block" — multiplies language diversity without multiplying demonstrations, which is valuable because collecting more physical episodes is far more expensive than generating more text. The risk is drift: an automatically generated paraphrase can add a detail the demonstration doesn't support, drop a constraint that mattered ("place it gently"), or introduce a reference to an object not in the scene. Any of these teaches the model a language-to-action association that's factually wrong.
Treat paraphrase generation as a draft step with the same verification requirement as any other automated labeling: generate candidates, then check each one against the episode — either with a human pass on a sample, or with a second model checking the paraphrase for added or dropped facts against the original — before it enters the training set unreviewed.
Capture-time vs. after-the-fact annotation
The cheapest and often most accurate instructions come from the operator, at capture time — typed or spoken immediately before or during the episode, describing the task they're about to do or are doing. This has a structural advantage over after-the-fact annotation: the person writing the instruction is also the person who just performed the action, so the instruction and the demonstration are unlikely to diverge.
After-the-fact annotation, done by a separate reviewer watching the recorded episode, costs more time per episode but can add structure the operator had no reason to produce live — sub-task boundaries, richer object descriptions, or corrections where the operator's live instruction didn't match what they ended up doing. The two aren't mutually exclusive: capture-time instructions as the baseline, with after-the-fact review for the sub-task segmentation or dense captioning that capture-time annotation can't produce.
VLM-assisted labeling with human verification
Vision-language models can draft instructions, sub-task boundaries, and even dense captions directly from episode video at a fraction of the cost of manual annotation from scratch, and for large datasets this is usually the only economically viable way to reach dense annotation coverage. The caveat is unconditional: a VLM-drafted label is a draft, not a ground-truth label, until a human has confirmed it against the video. VLMs confidently hallucinate objects, actions, and outcomes that aren't in the frame, and an unverified VLM label pipeline silently reintroduces exactly the kind of factually-wrong instruction that paraphrase augmentation risks — at much higher volume.
A workable split: use the VLM to draft, route every draft through a human reviewer whose job is confirm-or-correct rather than write-from-scratch (much faster per episode), and track what fraction of drafts the reviewer had to correct as a running quality signal on the VLM step itself.
Inter-annotator agreement
With more than one annotator, style guide drift is inevitable without a way to measure it. Give a shared sample of episodes to every annotator independently and compare. Exact string match is the wrong metric for free text — two annotators can both write a correct, differently-worded instruction — so measure agreement on structured sub-fields instead: the object referenced, the action verb, the target location or state. Disagreement concentrated in one sub-field usually points at an ambiguity in the style guide itself rather than an annotator being careless, and it is the fastest way to find out that "the red block" and "the block" are being used inconsistently for the same object well before it shows up as noisy grounding in a trained model. Pair this with the sampling approach in QA for demonstration datasets so language review rides the same reviewer pipeline as episode quality review, instead of running as a separate, easy-to-skip process.