Sensor Time Sync for Robot Learning: PTP, Triggers, Sub-ms

A search for "sensor time synchronization robot learning" turns up almost nothing but arXiv papers on multi-camera calibration and factor-graph SLAM. That's a gap, because the problem most teams hit isn't in a paper — it's a wrist camera free-running at 29.97 Hz against a controller loop ticking at exactly 500 Hz, and a dataset that quietly encodes a 30ms lag between what the gripper saw and what it did.

Policies trained on that dataset don't fail loudly. They fail as "flaky" grasp success rates no amount of extra demonstrations fixes, because the demonstrations are internally inconsistent. This is the practical reference we wished existed: clock taxonomy, why NTP isn't enough, how PTP and hardware triggering get you to microseconds, how to measure residual skew, and how to keep timestamps honest across drift and format conversion.

Why 30ms matters more than it sounds like it should

Imagine ACT or a diffusion policy trained on RGB + proprioception + force-torque, each stream stamped with time.time() on arrival. If the camera frame is captured 30ms before the joint state it's paired with, every "action" in the dataset is really the action taken 30ms after what the model saw — a systematic lag baked into the labels. At 10 Hz that's a third of a control period; at 50 Hz, more than one and a half steps.

Contact-rich tasks are worse: a peg-in-hole insertion generates a force spike lasting single-digit milliseconds. If your F/T sensor and vision stream disagree by even 10-20ms, contact events land in the wrong video frame — the policy learns to associate contact with the frame before or after the one where it actually happened. Multi-camera rigs compound this: two cameras individually "close enough" to the robot clock can be tens of milliseconds apart from each other, breaking stereo consistency and triangulation-based labeling.

The clock taxonomy nobody tells you about

A single robot rig usually has four distinct clocks in play, and none of them agree by default:

• Camera hardware clock — an internal oscillator incrementing on each exposure. Accurate relative to itself, meaningless relative to anything else unless disciplined.
• Host monotonic clock (CLOCK_MONOTONIC) — never jumps backward, immune to NTP step corrections, but has no defined epoch and is per-host. Two machines' monotonic clocks share no common reference.
• Wall clock (CLOCK_REALTIME) — has a real epoch (Unix time), the default choice, but steps forward or backward whenever NTP corrects it, silently reordering timestamps mid-capture.
• Robot controller clock — often a real-time kernel (Xenomai, PREEMPT_RT, a vendor RTOS) ticking at a fixed period, frequently isolated from the host's network stack entirely.

Sensor fusion and imitation learning implicitly assume these are one clock. They are not — the problem is bridging them with a bounded, known error instead of an unknown one.

NTP, PTP, and GPS/PPS: what each buys you

NTP's fundamental limit: it corrects for round-trip network delay statistically, over many packets, with no guarantee outbound and return paths are symmetric — hence the millisecond-scale floor. PTP fixes this by timestamping the sync packet in the NIC hardware, at the exact moment it hits the wire, removing OS scheduling jitter entirely. That's the difference between "close enough for logging" and "close enough for control."

# Minimal PTP setup on Linux for a camera + host on the same switch.
# Requires a PTP-capable NIC (check `ethtool -T eth0` for hardware timestamping support).
 
# 1. Run ptp4l to discipline the NIC's PHC (PTP Hardware Clock) via the network
sudo ptp4l -i eth0 -m --step_threshold=1 -f /etc/ptp4l.conf
 
# 2. Sync the host's system clock to that PHC with phc2sys
sudo phc2sys -s eth0 -c CLOCK_REALTIME -w -m
 
# 3. Verify offset is converging (want steady-state well under 10us)
sudo pmc -u -b 0 'GET CURRENT_DATA_SET'

If you can't get a hardware-timestamping NIC to every node, GPS+PPS is the alternative ground truth: a receiver disciplines a local oscillator to GPS time and emits a pulse-per-second edge any capture card can latch against — overkill for a bench arm, but standard for multi-robot fleets and outdoor rigs with no shared switch to run PTP over.

Hardware triggering beats software timestamps, full stop

For cameras specifically, the fix isn't a better clock — it's removing the ambiguity about when the shutter fired. Most machine-vision cameras (GigE Vision, USB3 Vision, many RealSense and industrial modules) expose a sync-in / trigger pin: drive it with a shared square wave (a microcontroller, an FPGA, or one camera as trigger master with a strobe output) and every camera on that line exposes at the same instant, sub-microsecond jitter between them.

• Software trigger (cam.capture() in a loop): timestamp reflects host scheduling, not the shutter — cross-camera skew is whatever the OS scheduler feels like.
• Hardware trigger, host-timestamped: cameras fire together, but you log arrival time, not exposure time — a roughly constant latency you should measure and subtract.
• Hardware trigger, PTP-stamped: the trigger pulse itself comes from a PTP-disciplined source, so exposure time lands in the same time base as everything else. The only option that gets sub-millisecond alignment without heroics.

If your rig can't do hardware triggering — many BYO setups and consumer webcams have no sync-in — measure the residual skew and correct for it instead.

Measuring the skew you still have

Two field-tested methods, no new hardware:

1. LED-flash test. Wire an LED into the frame of every camera, driven by a GPIO pulse timestamped against whatever clock you consider ground truth (ideally PTP-disciplined). Flash it once. The frame where the LED turns on, minus that camera's own timestamp, gives its fixed latency offset directly. Repeat if anything changes — cable swap, firmware update, USB hub reseat.

2. Cross-correlation of IMU vs. vision motion. Shake the sensor payload and extract a motion signal from both streams — angular velocity from the IMU, optical flow magnitude from vision. Cross-correlate the two series; the lag at peak correlation is your residual offset. This is the same technique visual-inertial odometry systems (VINS-Mono, OpenVINS) use for online IMU-camera calibration, borrowed here for measurement rather than fusion.

import numpy as np
from scipy.signal import correlate
 
def estimate_skew_ms(imu_gyro_mag, imu_ts, vis_flow_mag, vis_ts, target_hz=200):
    """Resample both signals to a common rate, cross-correlate, return lag in ms."""
    t0, t1 = max(imu_ts[0], vis_ts[0]), min(imu_ts[-1], vis_ts[-1])
    t_common = np.arange(t0, t1, 1.0 / target_hz)
    a = np.interp(t_common, imu_ts, imu_gyro_mag)
    b = np.interp(t_common, vis_ts, vis_flow_mag)
    a, b = (a - a.mean()) / a.std(), (b - b.mean()) / b.std()
    xcorr = correlate(a, b, mode="full")
    lags = np.arange(-len(b) + 1, len(a))
    best_lag = lags[np.argmax(xcorr)]
    return best_lag * (1000.0 / target_hz)

Run this at the start of a capture campaign and again periodically — any nonzero, stable skew should be baked into your ingestion pipeline as a per-sensor offset, not left for the policy to "figure out."

Drift over long sessions, and surviving format conversion

Even a well-disciplined clock drifts between corrections — cheap crystal oscillators run at tens of parts-per-million, so a free-running clock can wander a full millisecond every 10-30 seconds. Over a long teleop episode, one offset measured at t=0 isn't good enough. The standard fix is piecewise linear correction: measure offset at multiple points across the session and interpolate between them rather than assuming a constant bias. If you're PTP-disciplined throughout, this is moot — running ptp4l/phc2sys continuously means correction happens continuously, not as a single offset applied after the fact.

Format conversion is where sync work quietly gets thrown away. MCAP records both a log time (when the recorder observed the message) and a publish time (when the source node stamped it) — propagate publish_time downstream, not log_time, or you've reintroduced host-arrival jitter as ground truth. Same issue converting into LeRobot's episode format: it builds one per-episode timeline and resamples every modality onto it, so whatever timestamps you feed the converter become the training signal. ROS 2 bag → HDF5/Parquet conversions share the failure mode with header.stamp vs. bag-relative time.

We build Telemanual's capture pipeline around exactly this discipline — hardware-triggered where the rig supports it, PTP-disciplined otherwise, every exported format carrying original per-sensor timestamps rather than collapsed arrival times — because temporal integrity isn't something you can fix after the fact.