Sub-100ms Robot Teleoperation: A Latency Engineering Guide

Every teleoperation stack eventually runs into the same wall: it feels fine on a LAN demo and falls apart the moment the operator and the robot are on different continents. The reason is rarely bandwidth. It's the accumulation of small, unglamorous delays — a frame sitting in an encoder queue, a jitter buffer padding for network variance, a scheduler tick on the robot's control loop — that individually look negligible and collectively push you past the point where a human can close the loop on contact-rich manipulation.

This post is the latency budget we hold ourselves to, why 100ms end-to-end (glass-to-glass, not just network RTT) is the line that matters, and what actually moves the needle versus what just looks like it does on a benchmark.

Why 100ms, specifically

The number isn't arbitrary. Human sensorimotor literature on teleoperated manipulation (Sheridan's classic work on time-delay teleoperation, and more recent HRI studies on remote grasping) consistently finds that operators shift from continuous closed-loop control to "move-and-wait" behavior once round-trip delay crosses roughly 100-150ms. Below that threshold, hand-eye coordination during a peg-in-hole insertion or a force-controlled grasp stays fluid. Above it, operators unconsciously start pausing after each motion to let visual feedback catch up — throughput drops and error rates climb, especially on tasks requiring force feedback or fine contact (rigid part mating, deformable object handling).

For fleet data collection specifically, the cost isn't just discomfort: laggy teleop sessions produce worse demonstrations. Policies trained via imitation learning (ACT, diffusion policies, and their descendants) inherit the operator's hesitations, so a jittery 250ms session doesn't just annoy the operator, it degrades the training distribution.

The full glass-to-glass budget

Here's a representative budget for a robot in Frankfurt and an operator in the US, encoding 720p at 60fps over the public internet with a well-placed edge relay:

The video path and the command path share the same network legs, so they aren't purely additive in wall-clock terms — the operator perceives video latency and command latency as overlapping, not sequential. But every stage above is a place engineering effort either pays off or gets wasted, and each one has a failure mode that blows the budget by 2-5x if left untuned.

Why plain WebRTC isn't enough

WebRTC gets you 80% of the way for free: NAT traversal (ICE/STUN/TURN), SRTP encryption, and a congestion-controlled transport. But it was built for video calls, and video-call assumptions actively fight teleoperation:

• Adaptive bitrate fights control traffic. WebRTC's default congestion controller (GCC, or BBR in newer stacks) probes for bandwidth by occasionally overshooting and backing off. That's fine for a Zoom call where a dropped frame is invisible; it's a problem when a bandwidth probe collides with a time-critical command packet on the same congestion-controlled path.
• Jitter buffers default to comfort, not latency. Browsers' built-in RTCRtpReceiver jitter buffers are tuned to avoid visible stutter for human viewers watching passively, which means multi-frame buffering by default. You have to explicitly set playoutDelayHint (Chrome) or run your own jitter buffer to get it down to single-frame sizing.
• Video and data channels don't share priority by default. If your commands ride on an RTCDataChannel (SCTP) alongside a video MediaStreamTrack, they can queue behind video packets during congestion unless you explicitly separate them onto distinct transports or set DSCP/networkPriority hints.
• Simulcast and adaptive resolution add hidden latency. Encoders that renegotiate resolution mid-session to hit a bitrate target introduce keyframe requests, which cost a full frame interval or more whenever they fire.

None of this means "don't use WebRTC." It means treat it as a transport primitive you configure aggressively, not a latency solution you get by default.

What actually works

Hardware encode, tuned for latency over quality. NVENC (or equivalent VA-API/QuickSync paths) with a zero-latency preset, no B-frames, and a short GOP gets encode time down to single-digit milliseconds. A representative low-latency ffmpeg encode configuration:

ffmpeg -f v4l2 -i /dev/video0 \
  -c:v h264_nvenc -preset p1 -tune ll -zerolatency 1 \
  -rc cbr -b:v 6M -g 30 -bf 0 \
  -f rtp rtp://relay.internal:5004

-tune ll -zerolatency 1 disables lookahead and B-frame reordering — both of which trade latency for compression efficiency you don't need at this bitrate. -g 30 keeps keyframe intervals short so a dropped frame recovers within half a second, not several.

H.264 vs AV1 — pick H.264 for the live path. AV1 gives you 30-40% better compression at the same quality, which matters for storage and bandwidth-constrained links. But AV1 hardware encoders are still less universally available at low latency, and encode complexity is higher even in silicon. For the live control path we default to H.264 baseline/high profile with hardware encode; AV1 (or H.265) is worth it for the recorded/archival copy of the session where compression ratio matters more than encode latency.

Edge relays with anycast routing. A relay near the robot and another near the operator, connected over a private backbone or well-peered transit, consistently beats direct public-internet routing — public BGP paths are optimized for cost, not latency, and often take a scenic route between continents. Anycast lets the operator's connection always hit the topologically nearest POP without manual configuration.

Jitter buffers sized in single frames, not milliseconds-as-a-constant. A buffer should hold enough to absorb one frame interval of network jitter (at 60fps, ~16.7ms) plus a small safety margin, not a fixed 100ms buffer that "feels safe." Adaptive jitter buffers that shrink during stable network conditions and grow only when they observe actual variance keep the median latency low without sacrificing worst-case robustness.

Predictive streaming and local echo. For the command path specifically, echoing the operator's own input locally (rendering a ghost/predicted end-effector pose immediately, then reconciling with the robot's actual state once it arrives) removes the perceived round-trip from the operator's motor loop, even though the physical robot still moves on the real network schedule. This is the same trick flight sims and fighting games use for input lag — it doesn't reduce actual latency but it collapses perceived latency for the human closing the loop.

Measuring it honestly

Software-reported "latency" metrics lie constantly, because they measure the parts of the pipeline the software can see, not the parts a human experiences. Two disciplines fix this:

1. Measure glass-to-glass with hardware, not timestamps alone. A photodiode on the display paired with an LED triggered at capture time gives you a ground-truth measurement uncontaminated by clock skew between robot and operator machines. Where that's impractical, a millisecond timestamp overlay burned into the video frame at capture, read back visually or via OCR at the display, is a solid proxy — just make sure both ends share a PTP (IEEE 1588) synchronized clock or the overlay method just measures clock drift.
2. Report p95, not median. Median latency is a marketing number. The operator's experience is dominated by the bad tail — the one frame in twenty that arrives late enough to break the motor loop's rhythm. A system with a great median and a fat p95 tail feels worse than a system with a slightly higher median and a tight distribution.

We track both continuously in production, and gate any pipeline change on p95 glass-to-glass latency, not average. It's the only number that correlates with operators actually reporting the session as "responsive."

At Telemanual, this budget is why we hold median glass-to-glass latency under 80ms and stay sub-100ms over ordinary public internet — not a LAN, not a dedicated line, just a well-engineered path from camera to command and back.