A robot rig usually has no single clock. The camera has its own internal oscillator, the host computer has a wall clock that NTP nudges around, and the controller often runs on an isolated real-time clock with no defined relationship to either. Time synchronization is the work of bridging those clocks with a known, bounded error instead of an unknown one — and in robot data collection, it's one of the few problems that fails silently: an unsynchronized rig produces data that looks completely normal and trains a policy that just doesn't work very well, for reasons that don't show up in any single stream.
Why unsynchronized streams corrupt training data
Imitation learning assumes an observation-action pair actually happened together. If a camera frame is captured 30ms before the joint state it gets paired with in the dataset, every recorded "action" is really the action taken 30ms after what the model is told the robot saw — a consistent, systematic lag baked into every training example. At a 10Hz control rate that's a third of a control period; contact events in force-torque data can be off by enough to land in the wrong video frame entirely, which is especially damaging for demonstration data involving insertion or other brief, high-frequency contact events. Multi-camera rigs compound the problem: two cameras individually "close enough" to the robot's clock can still be tens of milliseconds apart from each other, which breaks any labeling or triangulation that assumes simultaneity.
Where the skew actually comes from
Most rigs have skew from more than one source at once:
- Driver timestamp versus capture time. A
cv2.VideoCaptureorcv_bridgetimestamp usually reflects when the frame was pulled from a driver buffer, not when the shutter opened. Under host load — a busy GPU, a full queue — that gap grows unpredictably rather than staying constant. - USB latency and buffering. Consumer USB cameras rarely deliver frames on a strict period; they buffer internally and hand frames to the OS in bursts, so "30fps" doesn't mean a frame every 33.3ms arrives on schedule.
- Free-running camera clocks. Without an external trigger, each camera's internal oscillator runs on its own schedule, so even two identical camera models drift apart from each other over a session.
- Wall-clock steps.
CLOCK_REALTIMEcan jump forward or backward whenever NTP corrects it mid-capture, silently reordering timestamps that were monotonic a moment before.
Getting clocks aligned: PTP, NTP, and hardware triggers
The tools available span a wide accuracy range, and the right one depends on what the recording actually needs:
- NTP / chrony synchronize wall clocks over ordinary networking, but correct for delay statistically across many packets under an assumption of symmetric path delay — good for coarse logging, not for aligning fast contact events, typically landing at one to tens of milliseconds on a LAN and worse over a WAN.
- PTP (IEEE 1588) synchronizes by timestamping sync messages in NIC hardware at the instant they hit the wire, removing OS scheduling jitter from the measurement. With a hardware-timestamping NIC this routinely reaches single-digit microsecond accuracy — the difference between "close enough for logging" and "close enough to trust for contact timing."
- gPTP (IEEE 802.1AS) is a restricted profile of PTP built for time-sensitive Ethernet networks. Where general PTP measures end-to-end delay, gPTP measures peer delay on every link independently, which suits multi-hop or multi-vendor sensor networks better than a single best-master-clock hierarchy does.
- Hardware trigger lines sidestep the clock-alignment problem for cameras specifically: a shared trigger pulse fired into every camera's sync-in pin makes them expose at the same physical instant, with the remaining work being to timestamp the trigger pulse itself against a trusted clock (ideally PTP-disciplined) rather than logging arrival time at each camera.
Approximate time synchronization at the software layer
Even with well-disciplined clocks, messages from different topics rarely arrive with identical timestamps — they're published at different rates and queued independently. ROS 2's message_filters package provides an ApproximateTimeSynchronizer that matches messages across topics by timestamp using an adaptive algorithm, within a configurable slop window, and hands the matched set to a callback as if they'd arrived together. This is a convenience for consuming already-reasonably-synced streams in a node — it does not fix a rig where the underlying skew is larger than the task can tolerate, and setting slop too loose just papers over misalignment that measurement would have caught.
Measuring the skew you actually have
Don't assume synchronization is working just because nothing looks obviously wrong — measure it directly, at the start of a collection campaign and again after any hardware change, since a cable swap or firmware update can move an offset without anything else appearing different. Two no-new-hardware methods cover most rigs: flashing an LED into every camera's frame against a trusted timestamp to read off each camera's fixed offset, and cross-correlating an IMU's motion signal against optical flow from vision to find the residual lag on a moving rig. The time-syncing-sensors field guide walks through both methods with working code and a full PTP setup, in more depth than a definition page needs.
What tolerance actually matters
The requirement scales with what the data needs to support rather than sitting at one fixed number. A reasonable working target is skew well under one control-loop period at your recording rate — at 10Hz that's under ~10ms, at 50Hz under ~2ms — since anything approaching a full control step starts systematically misattributing which observation caused which action. Contact-rich tasks with brief force transients need proportionally tighter alignment between the force-torque stream and vision than a slow pick-and-place task does.