Manual shape alignment

From the built shape model, we retrieved a mean shape (called source shape) which is used for initial alignment with target shape captured from Kinect.

Step 1: Scale source shape and target shape into same metric so only rigid transformation need to be considered. The shapes are shown in Figrue 1.

Figure 1 Target shapes (left) and source shapes (right)

Step 2: Select at least 3 key points correspondence at both shape manually. A larger number of correspondence is recommended as the error can be reduced. For offline processing, I select the key points using MeshLab as shown in the Figure 2.

Figure 2 Select key points correspondence in source shape (left) and target shape (right)

Step 3: According to the selected correspondence, we can use RANSAC method to estimate the 6 DoF rigid transformation. I also tried to use function estimateAffine3D() given by OpenCV, but the returned matrix is useless. The failure may caused by too few correspondences(3) were applied. Finally I get the transformation using MeshLab and the result is shown in Figure 3. In later stage, which combine 3D reconstruction and recognition, I will use the function in PCL to estimate the transformation and hope that works.

Figure 3 Result after two shapes are aligned based on the estimated transformation

Based on the alignment, I will fine adapt the source shape to the target shape with constraints from the statistical shape model.

Notes for VOSM fitting process

As I made the notes on papers, I scanned it and upload it as images for record.
Main fitting program begins:
It reads the input configuration and trained model for fitting.

For each testing image (2D), detect the key points position before try to fit model to target. The labeled function will be shown in a second.

 The function "VO_FirstEstimationBySingleWarp()" calculate transformation between model and target based on their corresponding key points. I also marked the OpenCV function for 3D case:

 Here is the place where the main fitting algorithm happens. Before trying to fit the target, we make sure the model after the transformation is within acceptable shape variance. The fitting contain several iteration for different image pyramid (scaled by 2, 4, 8 etc.). In my case, I will only use one pyramid (original size) since the model from Kinect and MRI are both in real human size.

The function "VO_CalcAllParams4AnyShapeWithConstrain()" project the current fitting model to the shape parameters which are constrained by the corresponding trained eigenvalues. These acceptable parameters are then back-projected to the original fitting shape using PCA.

For each image pyramid the function "PyramidFit()" try to update the current shape so that it is more and more similar to the target. As the update process may ruin the shape, the "VO_CalcAllParams4AnyShapeWithConstrain()" function is called again to prevent that happens. The fitting process will be terminated for two reasons: 1) Enough proportion of landmarks are fitted to the target; 2) Certain number of iteration has been went through so the fitting process won't go forever.

Now is the core function where the landmarks are moved to the target. As shown in the notes, each landmarks is moved to the best location according to matching profile. In my case, since the intensity is not available for the training data, I will discuss another strategy to find the best location of landmarks in future post.

Here is the last function, "VO_FindBestMatchingProfile1D". Although I will not find the best matching based on profile, it is still worth to investigate the idea.

To wrap up, the whole procedure can be refined as following flow chart:

Experiment of statistical shape model

In this post some artificial shapes by varying the shape parameters are presented. As the parameters are limited, the artificial shape is reasonable.
As shown in the Figure 1, the torso model is varied from fat to slim by changing the parameter corresponds to the largest eigenvector.

Figure 1 The shape variation as the changing parameter corresponds to the largest eigenvector (±3 standard deviation).

Figure 2 The shape variation corresponds to the second largest eigenvector (±3 standard deviation).

The second figure shows the breast variation which is achieved by changing the parameter corresponds to the second largest eigenvector.

Crop template model

As the torso model captured by Kinects doesn't has legs, arms and head. It's suggested that the models to be trained is better to be cropped. Therefore the trained model is more similar to the new torso model.
Since correspondences in the models given by Su-Lin are already existed, I can manually crop one model and leave the rest to program. Here I show the results after cropping.

Figure 1 Eight different models whose vertices were cropped (upper row: female; lower row: male)

Codes: PointFinder.h; PointFinder.cpp; ObjIO.h; ObjIO.cpp; main.cpp

Reading obj file

The training data contain mesh information of 3D body is stored in .obj file. Here is format of the .obj file:

####
#
# OBJ File Generated by Meshlab
#
####
# Object female01-1532-na.obj
#
# Vertices: 1282
# Faces: 2560
#
####
vn -0.344122 0.249172 -0.905258
v -155.144363 -54.513828 -640.656250
vn 0.046374 -0.339345 -0.939518
v -122.898865 -80.428886 -643.766418
...
vn 0.011051 0.127717 0.991749
v -53.384335 4.810349 852.756348
# 1282 vertices, 1282 vertices normals

f 5//5 1//1 16//16
f 29//29 18//18 6//6
...
f 545//545 501//501 503//503
# 2560 faces, 0 coords texture

# End of File
The lines start with '#' in file can be treated as comments as "//" in C/C++.
The lines begin with vn indicate that the three numbers is unit normal at the point. The next line starting with v indicate the corresponding vertex coordinate.
The lines starting with f indicate the face/mesh information. For example, the first triangle consist of vertex 5, 1 and 16.

Following is snippet for reading obj file and save the data as plain text.

// ObjReader.h
#pragma once

#include 
#include 
#include 
#include 
#include 
#include 

class CObjReader
{
public:
	typedef struct point3fTag
	{
		float x, y, z;
		float normx, normy, normz;
	}point3f;

	int m_iNbofVertex;
	CObjReader(void);
	~CObjReader(void);

	static void ReadObj(const std::string& filename, CObjReader &currObj);
};

// ObjReader.cpp
#include "ObjReader.h"

CObjReader::CObjReader(void):m_iNbofVertex(0){}

CObjReader::~CObjReader(void){}

void CObjReader::ReadObj(const std::string& filename, CObjReader &currObj)
{
	std::fstream fp;
	std::ofstream op(".\\output.txt");
	fp.open(filename.c_str(), std::ios::in);

	std::stringstream ss;
	std::string temp;
	float tempFloat = 0.0f;
	std::vector vertice;
	std::vector face;
	// Skip header
	do 
	{
		getline(fp, temp);
	} while (temp[0] == '#');

	// Process vertex and its normal
	while (temp[0] != '#')
	{
		ss << temp;
		ss >> temp;	// skip "vn"
		point3f tempPoint;
		ss >> tempPoint.normx >> tempPoint.normy >> tempPoint.normz;
		ss.clear();
		// get v
		getline(fp, temp);
		ss << temp;
		ss >> temp;	// skip 'v'
		ss >> tempPoint.x >> tempPoint.y >> tempPoint.z;
		ss.clear();
		vertice.push_back(tempPoint);
		// get vn for next point
		getline(fp, temp);
	}
	ss << temp;
	ss >> temp;	// skip '#'
	unsigned int a;
	ss >> a;
	ss.clear();
	assert(a == vertice.size());
	currObj.m_iNbofVertex = vertice.size();
	// Process face
	getline(fp, temp); // skip empty line
	getline(fp, temp); // Get first line of face data
	do 
	{
		ss.str("");	// Clean previous contents
		ss << temp;
		ss >> temp; // skip 'f'
		for (int i = 0; i < 3; i++)
		{
			ss >> temp; // temp: i//i
			temp.erase(temp.find('/'),temp.size()-temp.find('/'));
			face.push_back(std::stoi(temp));
		}
		ss.clear();
		getline(fp, temp);
	} while (temp[0] == 'f');
	std::cout << temp << std::endl;
	fp.close();
	
	// Save vertice and face info
	for (unsigned int i = 0; i < vertice.size(); i++)
	{
		op << "vn: ";
		op << vertice[i].normx << " " << vertice[i].normy << " " << vertice[i].normz << "\n";
		op << "v: ";
		op << vertice[i].x << " " << vertice[i].y << " " << vertice[i].z << "\n";
	}
	op << "# Face data\n";
	for (int i = 0; i < face.size();)
	{
		op << face[i] << " " << face[i+1] << " " << face[i+2] << "\n";
		i = i + 3;
	}
	op.close();
}

Notes of VOSM shape model building framework

The information saved for fitting:

\ShapeModel\
ShapeModel.txt:
m_iNbOfSamples
m_iNbOfShapeDim
m_iNbOfPoints
m_iNbOfShapes
m_iNbOfEigenShapesAtMost
m_iNbOfShapeEigens
m_iNbOfEdges
m_iNbOfTriangles
m_fAverageShapeSize
m_fTruncatedPercent_Shape
m_PCAAlignedShapeMean.txt
m_PCAAlignedShapeEigenValues.txt
m_PCAAlignedShapeEigenVectors.txt

m_VOAlignedMeanShape.txt

m_VOReferenceShape.txt // Reference shape which is scaled back to the original size and translated to origin

m_vShapes.txt // All loaded shapes in a vector format

m_vAlignedShapes.txt // All aligned shapes in a vector format

m_vShape2DInfo.txt // shape information and face parts information

m_vEdge.txt

m_vTemplateTriangle2D.txt // Unnormalised triangles in the template shape

m_vNormalizedTriangle2D.txt // Normalised triangles in the template shape

.\Point2DDistributionModel\

Point2DDistributionModel.txt // number of points

m_VONormalizedEllipses.txt 

VOSM 2D model building and fitting for face data

After successful compilation of VOSM we got "test_smbuilding.exe" and "test_smfitting.exe" which are responsible for model building and fitting respectively.

Preparation of database:

From the VOSM website we can download several database: link. For this post, I will use JIAPEI database. Within JIAPEI directory, there are two directories: .\annotations and .\images. The former one contains annotation of landmarks for both training and testing images. The later one contains the corresponding images. For each database there should be a ShapeInfo.txt which describes how the landmarks are connected as shape (with several parts).