Calibration of Kinect IR camera

Since the sdk function NuiImageGetColorPixelCoordinatesFromDepthPixelAtResolution gives pixel coordinates in color space that correspond to the specified pixel coordinates in depth space, I use IR camera (depth space) as reference.
I used Bouguet's MATLAB calibration toolbox to calibrate the IR camera individually and then find out extrinsic parameters between them. In this post I will focus on result and discussion, a comprehensive and completed step-by-step tutorial of the toolbox can be found in the website.
Before  taking IR images for the calibration I turned off the IR emitter by SDK function (NuiSetForceInfraredEmitterOff(TRUE)) so that no IR pattern would affect quality of the IR images. The first and second cameras are called the first and second camera respectively. For both camera, I chose 12 different IR images which represent different pose of calibration pattern, as shown in Figure 1.

Figure 1 12 calibration images

Let's start with the left IR camera. After reading the images into MATLAB, we need to extract the corners of the pattern. It can be done by selecting four corners of the calibration pattern in each image and the toolbox can detect the corner within the selected area automatically, as shown in Figure 2. In terms of selecting four boundary points, I would suggest to select the inner points rather than four boundaries of the pattern. In this case, the toolbox can detect the corners better so makes our life easier for later re-computation stage.

Figure 2 Grid corners extraction

After all grid corners have been extracted, we can visualise the extrinsic results (Figure 3) and reprojection error.

Figure 3 Extrinsic parameters (camera-centred)

The  intrinsic parameters of the left camera can be computed with uncertainties as shown below. More uncertainties, more error existed. In order to minimise the error, we can recompute the parameters by choosing the images that occur most error in the error diagram and extract the corners in a smaller and less distorted area. As shown in Figure 4, the corners are extracted within a 5x5 area instead of the original 5x7 area in the selected images.

Calibration results after optimization (with uncertainties):
Focal Length:          fc = [ 569.92842   571.44089 ] ± [ 16.23688   14.06564 ]
Principal point:       cc = [ 278.08552   258.82575 ] ± [ 25.86257   24.07492 ]
Skew:             alpha_c = [ 0.00000 ] ± [ 0.00000  ]   => angle of pixel axes = 90.00000 ± 0.00000 degrees
Distortion:            kc = [ -0.28627   0.90685   0.00121   -0.01097  0.00000 ] ± [ 0.18908   1.79441   0.01032   0.00938  0.00000 ]
Pixel error:          err = [ 0.28284   0.62366 ]

Figure 4 Grid corner extraction for re-calibration

Next, we can see the error became smaller than the first calibration results:

Calibration results after optimization (with uncertainties):
Focal Length:          fc = [ 581.12084   581.34563 ] ± [ 10.53285   9.09579 ]
Principal point:       cc = [ 281.39427   253.27383 ] ± [ 16.64980   15.86515 ]
Skew:             alpha_c = [ 0.00000 ] ± [ 0.00000  ]   => angle of pixel axes = 90.00000 ± 0.00000 degrees
Distortion:            kc = [ -0.26651   0.92108   0.00301   -0.01151  0.00000 ] ± [ 0.13097   1.44999   0.00626   0.00590  0.00000 ]
Pixel error:          err = [ 0.22538   0.35812 ]

By doing the procedures over and over again until an acceptable result has been achieved:

Calibration results after optimization (with uncertainties):

Focal Length:          fc = [ 579.39816   580.78490 ] ± [ 9.58715   8.48105 ]

Principal point:       cc = [ 279.82800   257.11149 ] ± [ 14.13074   12.45446 ]

Skew:             alpha_c = [ 0.00000 ] ± [ 0.00000  ]   => angle of pixel axes = 90.00000 ± 0.00000 degrees

Distortion:            kc = [ -0.32611   1.61254   0.00131   -0.01085  0.00000 ] ± [ 0.10937   1.22361   0.00600   0.00512  0.00000 ]

Pixel error:          err = [ 0.21153   0.29514 ]

For the right camera, the procedures are exactly same but one thing need to be noted is that the extracted corners in the left camera must correspond to the extracted corners in the right camera. Therefore, we need to be careful to the sequence when choosing the boundary of the calibration pattern in the images captured by the right camera.

Finally, the extrinsic parameters or in another word the relative position and orientation between two IR camera can be computed based on the individual calibration results. The extrinsic results is:


Extrinsic parameters (position of right camera wrt left camera):
Rotation vector:             om = [ 0.12700   1.83078  2.45355 ]
Translation vector:           T = [ 37.16528   -993.13499  676.55744 ]


The om vector is rotation vector which can be related to rotation matrix by Rodrigues' rotation formula. The three elements in the translation vector should be the rigid translation in x, y and z direction with unit of millimetre (mm). From Figure 5 we can visualise the relation between the left, right camera and the calibration pattern.

Say we have a point in the world and known its coordinate (X_L) in left camera reference frame, its coordinate (X_R) in the right camera reference frame can be calculated by:

X_R = R * X_L + T

Figure 5 Extrinsic parameters of stereo calibration