Kinect RGB & depth camera calibration

As mentioned in previous post, Kinect has two cameras, one for capturing colour image and the other for depth image. These two images don't match with each other (i.e two pixels at the same location in RGB and depth image do not correspond to the same location in a scene). In order to get the depth information of the colour image (or the colour information for depth image), we need to calibrate the two camera.
The idea of this calibration (and images) is come from here.

Step 1: Capture IR image (not depth) and colour images by using libfreenect (update: the newest Kinect for Windows SDK 1.6 also provide raw IR image). We need to block the IR emitter to get clean image instead of infrared spots for better corner detection in calibration. If the IR image is dark which means not enough IR rays are reflecting from the chessboard pattern, the Halogen lamp (heat) to illuminate the scene.

IR image (IR emitter is blocked) and Colour image captured by the corresponding camera

Step 2: w.r.t each camera, use calibration toolbox (e.g. GML) to get intrinsic parameters, K_ir and K_rgb and distortion parameters.

Step 3: Use toolbox to calculate extrinsic parameters of two cameras and get their relative transformation by using this equation:

where is the transformation from a to b. Since we already known the transformation from world coordinate frame to RGB and IR camera frame (extrinsic parameters), the relative position between IR and RGB camera can be found.

Step 4: Compute depth of RGB image from depth image provided by IR camera.

• Step 4-1: for each pixel p_ir in IR image, back-project it to 3D world then get 3D point P_ir:

P_ir = inv(K_ir) * p_ir

• Step 4-2: transform the 3D point from IR coordinate frame to RGB coordinate frame:

P_rgb = R*P_ir + t

• Step 4-3: project the 3D point to RGB image coordinate:

p_rgb = K_rgb*P_rgb

The depth of the pixel p_rgb is the Z value of P_rgb.

P_ir : 3D point in the IR camera's coordinate system
R, t : Relative transformation between two cameras
P_rgb : 3D point in the RGB camera's coordinate system
p_rgb : The projection of P_rgb onto the RGB image

Base on the same idea, the colour information of the depth image can be calculated in the same way.

Notes:
The depth images provided by the IR camera express the depth information as discretized values in certain range, rather than actual distances. The depth-to-distance is required to convert the raw depth values into real distances. The calibration process is straightforward. The Windows SDK or other libraries also provide actual distance but the precision may not good enough. The depth-to-distance calibration will be done depends on the accuracy of the library during experiment.