Collecting Contact-Rich Manipulation Data (2026 Guide)

Peg-in-hole, connector mating, cable routing, snap-fit assembly — the tasks that matter on a factory floor are the ones where a policy trained on RGB video alone falls apart the instant the gripper makes contact. We've watched this happen enough times to stop being surprised by it: a vision-only ACT policy nails the reach and approach, then stalls or grinds at the exact moment insertion begins, because the signal that disambiguates "aligned and sliding in" from "jammed against the chamfer" was never in the training data.

This is a guide to collecting the data that fixes that — force, tactile, and joint torque streams captured at the rates and alignment precision contact events actually require, not the rates convenient for a webcam.

Why vision-only demonstrations fail at contact

The core problem is occlusion combined with bandwidth. At the moment a peg enters a hole, the gripper, the part, and the fixture overlap in the camera frame — exactly when the informative signal is highest, the visual signal is lowest. A 30 Hz RGB stream also doesn't carry the information: insertion force profiles for a tight-tolerance fit change meaningfully over 5-10ms as the part contacts a chamfer, binds, and releases. You cannot recover that from frames spaced 33ms apart, no matter how good the encoder is.

The result is that vision-only policies learn a proxy: pose trajectories that worked for one instance of contact geometry and friction, with no mechanism to detect and correct for the binding, jamming, or misalignment a slightly different starting pose produces. Force and tactile observations generalize across those variations because they measure the physical interaction directly instead of inferring it from geometry.

The sensor stack

Three sensor classes cover most contact-rich manipulation work, and they answer different questions:

• 6-axis force/torque (F/T) at the wrist — ATI Nano/Mini series, Robotiq FT 300, or the integrated F/T in newer arms (e.g., Franka's torque-sensing joints estimating wrist wrench). Gives you the aggregate force and moment the end effector is exerting, which is what most insertion and assembly force profiles are defined against.
• Tactile arrays — two families with different trade-offs. GelSight-style optical sensors (GelSight Mini, DIGIT) give high-resolution deformation images (a camera looking at a gel pad), excellent for slip detection and fine contact geometry but expensive to process at high rate. Taxel-grid resistive/capacitive arrays (e.g., Xela uSkin) give sparser per-cell normal/shear force at lower resolution but much lower latency and bandwidth per sample.
• Joint torque sensing — series elastic actuators or torque-sensing joints (Franka, KUKA LBR iiwa) give a redundant, whole-arm view of contact forces, useful for detecting contact anywhere along the arm and for cross-checking the wrist F/T reading.

None of these substitute for the others. A wrist F/T sensor won't tell you about slip at the fingertip; a fingertip tactile array won't tell you the arm brushed a fixture on the way in.

Capture rates that actually matter

This is where teams most often under-provision. Video habits (10-30 Hz) do not transfer to force data.

Time alignment: the part everyone underestimates

Once you're mixing a 30 Hz camera, a 200 Hz F/T sensor, a 500 Hz tactile array, and a 1000 Hz joint state stream, the hard problem stops being "collect the data" and becomes "prove these streams agree on what time it is." A contact event that resolves in 15ms is smaller than one video frame interval — if sensor clocks drift by even a few milliseconds relative to each other, you can mislabel which frame the contact actually started in.

We sync to a single hardware or PTP (IEEE 1588) clock domain wherever sensors support it, and timestamp every sample at acquisition rather than at receipt by the logging process — USB and network stacks introduce their own jitter unrelated to when the sensor actually measured anything. Where PTP isn't available (many tactile sensors are USB-HID with no hardware timestamping), we cross-correlate against a shared reference signal — a physical tap or force impulse visible in every stream — to estimate and correct clock offset in post-processing.

import numpy as np
 
def align_streams(ref_ts, ref_signal, other_ts, other_signal, search_window_s=0.05):
    """Estimate the time offset between two sensor streams by
    cross-correlating a shared transient (e.g. a contact impulse)
    and resampling onto the reference clock."""
    dt = np.median(np.diff(ref_ts))
    lags = np.arange(-search_window_s, search_window_s, dt)
    scores = [
        np.corrcoef(
            ref_signal,
            np.interp(ref_ts + lag, other_ts, other_signal)
        )[0, 1]
        for lag in lags
    ]
    best_offset = lags[int(np.argmax(scores))]
    aligned = np.interp(ref_ts + best_offset, other_ts, other_signal)
    return aligned, best_offset

Operator technique for teleop demonstrations

Contact-rich demos are also harder to teleoperate well than free-space reaching, and operator technique shows up directly in downstream policy quality.

• Force feedback matters more here than anywhere else in teleop. An operator driving a leader-follower arm pair with force reflection (or basic haptic feedback on a device like a 3D Systems Touch or Force Dimension omega) can feel the chamfer contact and modulate insertion force in real time, producing adaptive, force-responsive demonstrations. Without force feedback, operators drive on vision alone and tend to either ram parts (risking damage, producing low-quality contact data) or hover cautiously above the surface, never completing the informative part of the task.
• Leader-follower arm setups (a kinematically similar leader driving a follower, as in ALOHA-style rigs) preserve force and motion fidelity better than joystick or 6-DoF mouse teleop for insertion-class tasks, because the operator's proprioception maps naturally onto the follower's joint torques.
• Gloves and vision-based hand tracking lose force fidelity. They capture hand pose and finger configuration well but don't close the loop on contact force — the operator guesses at grip force from visual feedback alone, producing far more force variance on a repeated task than leader-follower teleop does.

Labeling contact events

Manual labeling of contact onset/release across thousands of demonstrations doesn't scale, so we auto-detect contact events from the F/T signature: a derivative threshold on force magnitude (a few newtons over 2-3 samples at 200 Hz) flags onset; a return below threshold with a stable low-force window flags release. Each phase of a multi-step task (approach, contact, insertion, seating) gets its own success/failure label, derived from the force profile shape (a seated connector shows a characteristic plateau; a failed insertion shows oscillation or an early, lower-force plateau) plus terminal state checks (joint position, vision-based part pose if available).

This per-phase labeling is what lets you filter datasets down to successful contact phases for policy training, or mine failures for recovery-behavior fine-tuning.

How policies consume this data

Force-conditioned observations feed into the same architectures used for vision, with the force/tactile stream added as another observation modality. ACT (Action Chunking with Transformers) and diffusion policies both accept multi-modal observation encoders — a common pattern is a small MLP or 1D-conv encoder over the windowed F/T signal, concatenated with the vision backbone's features before the transformer or diffusion U-Net. The force observation window needs to match the timescale of the phenomenon: a single 200 Hz sample tells the policy almost nothing about a chamfer slide; a 100-200ms window (20-40 samples) gives it the shape of the transient.

Storage is the other place high-rate force data breaks naive pipelines — a 500 Hz, 6-axis float32 F/T stream is roughly 12 KB/s per episode, trivial alone but it multiplies fast across a fleet and needs to stay time-indexed against video and joint state rather than living in a separate CSV nobody re-syncs. We store aligned multi-modal episodes in whichever of

Telemanual's capture pipeline records F/T, tactile, and joint torque alongside video for every teleoperated session, time-aligned at acquisition and exported directly into these formats — so the contact data described above is a byproduct of normal data collection, not a separate integration project.