Point clouds generation and alignment for two Kinects

Finally, they are aligned.

In this post, I will introduce coloured point cloud generation based on depth (disparity) and RGB images captured by IR and RGB camera respectively. Afterwards, the point clouds generated from two Kinects are aligned together based on the stereo calibration results from previous post: link.

- Point cloud generation
As the colour camera and depth camera are not at the same location and the field of view are different, the depth frame and colour frame are not aligned together without transformation. It is easy to implement the alignment between the colour and depth frame by using Kinect SDK which provides methods to transform the frame from one to the other. Here is the procedures with the API I used:

i. Given pixel coordinate in depth image, use this function NuiImageGetColorPixelCoordinatesFromDepthPixelAtResolution to get its corresponding pixel coordinate (x, y) in colour image.
ii. If the pixel (x, y) exceeds the boundary of the colour image, then set this point to nothing (located at origin without colour). Otherwise, colour the depth pixel.
iii. Given pixel coordinate in depth image, use NuiTransformDepthImageToSkeleton to get its 3D coordinate in skeleton space. Based on SDK document, the skeleton system mirrors the scene so a person facing to Kinect is considered to be looking in the -z direction in skeleton space. Here we need to invert x value in order to get normal non-inverted image.
iv. The function NuiTransformDepthImageToSkeleton returns a 4 elements vector which is represented as [x, y, z, w]. In order to get the real coordinate of the pixel (point), we can divide the first three elements (x, y and z) by the fourth elements that is: X = x/w, Y = y/w and Z = z/w.

- Point cloud alignment
It is a straightforward problem if we can correctly calibrate the extrinsic parameters and clearly define the coordinate systems. One thing I got stuck for long time is the different coordinate definition for Kinect. As shown in Figure 1, the SDK defines right-handed coordinate system that places a Kinect at the origin with the positive z-axis extending in the direction in which the Kinect is pointed. The positive y-axis extends upward, and the positive x-axis extends to the left. However, the MATLAB calibration toolbox also uses right-handed system but rotated around Z axis for 180 degrees from skeleton coordinate.

Figure 1 Kinect skeleton coordinate (blue) and camera coordinate (red) in MATLAB calibration toolbox.

Therefore, if we use the calibration result from the MATLAB toolbox we need to transform the point from skeleton coordinate to MATLAB camera coordinate and transform it back after the transformation from one Kinect coordinate to another has finished.

Figure 2 An animation for the point cloud alignment result.

As shown in the Figure 2, the result of alignment is promising but the interference problem still existed.