A multi-camera robot rig has three separate calibration problems stacked on top of each other, and it's easy to solve one and assume the rig is "calibrated." Intrinsics describe each camera's own lens geometry. Extrinsics describe how the cameras relate to each other. Hand-eye calibration links the cameras to the robot's kinematic frame, which is the transform that actually matters for anything touching end-effector pose. Skipping or half-doing any one of them produces a rig that looks fine in a video feed and quietly corrupts every derived pose in the dataset — the kind of silent failure a QA pass on the resulting episodes won't catch, because nothing about a miscalibrated rig looks wrong in the video itself.
Calibrate each camera's intrinsics
Intrinsics — focal length, principal point, and lens distortion coefficients — are per-camera and independent of the rig. The standard target is a ChArUco board: a checkerboard pattern overlaid with ArUco markers, which lets OpenCV's cv2.aruco.CharucoDetector identify board corners even under partial occlusion, unlike a plain checkerboard where a single blocked corner can break detection.
import cv2
import numpy as np
board = cv2.aruco.CharucoBoard((7, 5), 0.035, 0.026, cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_5X5_100))
detector = cv2.aruco.CharucoDetector(board)
all_corners, all_ids, image_size = [], [], None
for img_path in image_paths:
img = cv2.imread(img_path)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
charuco_corners, charuco_ids, _, _ = detector.detectBoard(gray)
if charuco_corners is not None and len(charuco_corners) > 4:
all_corners.append(charuco_corners)
all_ids.append(charuco_ids)
image_size = gray.shape[::-1]
ret, camera_matrix, dist_coeffs, rvecs, tvecs = cv2.aruco.calibrateCameraCharuco(
all_corners, all_ids, board, image_size, None, None
)
print(f"RMS reprojection error: {ret:.4f} px")Capture 20–40 images per camera covering the full frame — corners and edges, not just the center — and a range of distances and tilt angles. A board held only flat and centered under-constrains the distortion model and gives a deceptively low error that doesn't hold up off-axis.
Calibrate extrinsics between cameras
Extrinsics answer "where is camera B relative to camera A." With the same board visible in overlapping camera views simultaneously, cv2.stereoCalibrate solves for the rotation and translation between a camera pair given each camera's already-known intrinsics. For a rig with more than two cameras, calibrate each camera against one designated reference camera (or against the robot base directly, if every camera has a workspace view) rather than chaining pairwise transforms, which accumulates error with every hop.
How much this matters scales with what the cameras are doing together. A rig with one wrist camera and one static overview camera that are never fused into a single 3D estimate can tolerate looser extrinsics than a stereo pair used for depth, or a three-plus camera array feeding a triangulation-based keypoint or pointcloud pipeline. If every camera only ever contributes its own 2D pixels to the policy independently, extrinsic error mostly shows up as reduced cross-view consistency rather than a hard failure — still worth fixing, but not the first thing to chase if the rig is otherwise stable.
Run hand-eye calibration
This is the step that's easy to skip because the rig "looks" calibrated without it — and the one that actually matters for anything using end-effector pose. Hand-eye calibration solves for the fixed transform between a camera frame and the robot's kinematic frame — the same frame tree your URDF defines — by pairing target poses observed by the camera with the corresponding end-effector poses reported by the robot, across a sequence of moves. This is the classic AX = XB formulation: X is the unknown camera-to-gripper (or camera-to-base) transform, and A and B are the relative motions observed by the robot and the camera respectively between two poses.
Two rig configurations, two setups of the same math:
- Eye-in-hand. Camera mounted on the moving end-effector, observing a static target fixed in the workspace. Solves for the camera-to-gripper transform.
- Eye-to-hand. Camera mounted statically in the workspace, observing a target attached to the moving end-effector. Solves for the camera-to-base transform.
cv2.calibrateHandEye implements several published solutions to AX = XB — Tsai–Lenz, Park, Horaud, Andreff, and Daniilidis — selectable by method flag. They generally agree closely on well-conditioned data and diverge more on noisy or poorly-varied pose sets, so it's worth running more than one method and comparing.
R_cam2gripper, t_cam2gripper = cv2.calibrateHandEye(
R_gripper2base, t_gripper2base, # from robot forward kinematics, one per pose
R_target2cam, t_target2cam, # from the intrinsics-calibrated camera, same poses
method=cv2.CALIB_HAND_EYE_PARK,
)Collect 15–20 robot poses with meaningfully varied rotation — pure translation between poses under-constrains the rotational part of the solve, which is the most common reason hand-eye calibration produces a plausible-looking but wrong answer.
If neither the robot base nor the calibration target sits in a coordinate frame you know in advance — for example, a static camera watching a target held by the arm on a mobile base whose position relative to the workspace also isn't fixed — the two-transform AX = XB model isn't sufficient. OpenCV's cv2.calibrateRobotWorldHandEye solves the more general robot-world/hand-eye problem, recovering both the camera-to-robot-base transform and the target-to-world transform simultaneously. Reach for it only when you actually have two unknown frames; it needs more data to converge reliably than the standard formulation and is easy to misapply.
Validate with reprojection error
Reprojection error — the pixel distance between a detected calibration point and where the solved model predicts it should land — is the primary sanity check at every stage. For intrinsics, well under 1 pixel RMS on a sharp, well-lit board is a reasonable bar; consistently higher suggests a bad board print, motion blur in the capture set, or too little viewpoint variation. For hand-eye calibration, project the target's known geometry through the solved transform chain and check the residual the same way — and check that error is evenly distributed across the image and across poses, not concentrated at the edges or on a subset of captures, which usually points at a specific bad sample rather than a systematic model problem.
Record the calibration as versioned dataset metadata
Calibration parameters belong next to the data they apply to, not in a wiki page or a config file someone has to remember to check. Store intrinsics, extrinsics, and hand-eye transforms as structured metadata attached to the episodes recorded under that calibration — tagged with a calibration ID, the date it was run, and the reprojection error achieved — so a training pipeline (or anyone auditing a dataset months later) can tell exactly which geometry applied to which episode without cross-referencing a separate document that may or may not still be accurate. On a ROS 2 stack it's tempting to treat the published static transform tree as the calibration record — it isn't, on its own, because a tf publisher doesn't carry the reprojection error or the calibration date that tell you whether the transform is still trustworthy.
- STORE
- Camera matrix, distortion coefficients, extrinsic and hand-eye transforms per calibration ID
- TAG
- Every episode with the calibration ID active when it was recorded
- VERSION
- Never overwrite a calibration record — append a new ID and supersede
Drift, recalibration cadence, and what miscalibration does downstream
Cameras and mounts drift: thermal expansion shifts a lens slightly, a cable snag nudges a mount, a hard stop on the arm transmits a shock through the whole rig. Recalibrate immediately after any disassembly, collision, or mount adjustment — that part isn't optional. Absent an incident, run a lightweight periodic drift check — reproject a known reference point and confirm the error hasn't grown — rather than trusting a fixed calendar interval, since mechanical stability varies enormously between a rig bolted to a welded frame and one that lives on a cart.
The failure mode that makes this worth the discipline is that a drifted calibration doesn't announce itself. A policy trained on data with a slowly drifting camera-to-base transform learns a systematically biased mapping from pixels to action, and that shows up at evaluation time as degraded grasp accuracy or contact timing with no obvious cause — because nothing in the training loss reflects a geometric error baked into the labels. Treating calibration as versioned, dated metadata rather than a one-time setup step is what makes that failure traceable instead of mysterious.