24 May 2013

Notes for VOSM fitting process

As I made the notes on papers, I scanned it and upload it as images for record.
Main fitting program begins:
It reads the input configuration and trained model for fitting.

For each testing image (2D), detect the key points position before try to fit model to target. The labeled function will be shown in a second.
 The function "VO_FirstEstimationBySingleWarp()" calculate transformation between model and target based on their corresponding key points. I also marked the OpenCV function for 3D case:

 Here is the place where the main fitting algorithm happens. Before trying to fit the target, we make sure the model after the transformation is within acceptable shape variance. The fitting contain several iteration for different image pyramid (scaled by 2, 4, 8 etc.). In my case, I will only use one pyramid (original size) since the model from Kinect and MRI are both in real human size.


The function "VO_CalcAllParams4AnyShapeWithConstrain()" project the current fitting model to the shape parameters which are constrained by the corresponding trained eigenvalues. These acceptable parameters are then back-projected to the original fitting shape using PCA.

For each image pyramid the function "PyramidFit()" try to update the current shape so that it is more and more similar to the target. As the update process may ruin the shape, the "VO_CalcAllParams4AnyShapeWithConstrain()" function is called again to prevent that happens. The fitting process will be terminated for two reasons: 1) Enough proportion of landmarks are fitted to the target; 2) Certain number of iteration has been went through so the fitting process won't go forever.

Now is the core function where the landmarks are moved to the target. As shown in the notes, each landmarks is moved to the best location according to matching profile. In my case, since the intensity is not available for the training data, I will discuss another strategy to find the best location of landmarks in future post.
Here is the last function, "VO_FindBestMatchingProfile1D". Although I will not find the best matching based on profile, it is still worth to investigate the idea.
To wrap up, the whole procedure can be refined as following flow chart:

16 May 2013

Experiment of statistical shape model

In this post some artificial shapes by varying the shape parameters are presented. As the parameters are limited, the artificial shape is reasonable.
As shown in the Figure 1, the torso model is varied from fat to slim by changing the parameter corresponds to the largest eigenvector.
Figure 1 The shape variation as the changing parameter corresponds to the largest eigenvector (±3 standard deviation).
Figure 2 The shape variation corresponds to the second largest eigenvector (±3 standard deviation).
The second figure shows the breast variation which is achieved by changing the parameter corresponds to the second largest eigenvector.


14 May 2013

Crop template model

As the torso model captured by Kinects doesn't has legs, arms and head. It's suggested that the models to be trained is better to be cropped. Therefore the trained model is more similar to the new torso model.
Since correspondences in the models given by Su-Lin are already existed, I can manually crop one model and leave the rest to program. Here I show the results after cropping.
Figure 1 Eight different models whose vertices were cropped (upper row: female; lower row: male)

Codes: PointFinder.hPointFinder.cppObjIO.hObjIO.cppmain.cpp

09 May 2013

Reading obj file

The training data contain mesh information of 3D body is stored in .obj file. Here is format of the .obj file:

####
#
# OBJ File Generated by Meshlab
#
####
# Object female01-1532-na.obj
#
# Vertices: 1282
# Faces: 2560
#
####
vn -0.344122 0.249172 -0.905258
v -155.144363 -54.513828 -640.656250
vn 0.046374 -0.339345 -0.939518
v -122.898865 -80.428886 -643.766418
...
vn 0.011051 0.127717 0.991749
v -53.384335 4.810349 852.756348
# 1282 vertices, 1282 vertices normals

f 5//5 1//1 16//16
f 29//29 18//18 6//6
...
f 545//545 501//501 503//503
# 2560 faces, 0 coords texture

# End of File
The lines start with '#' in file can be treated as comments as "//" in C/C++.
The lines begin with vn indicate that the three numbers is unit normal at the point. The next line starting with v indicate the corresponding vertex coordinate.
The lines starting with f indicate the face/mesh information. For example, the first triangle consist of vertex 5, 1 and 16.

Following is snippet for reading obj file and save the data as plain text.
// ObjReader.h
#pragma once

#include 
#include 
#include 
#include 
#include 
#include 

class CObjReader
{
public:
	typedef struct point3fTag
	{
		float x, y, z;
		float normx, normy, normz;
	}point3f;

	int m_iNbofVertex;
	CObjReader(void);
	~CObjReader(void);

	static void ReadObj(const std::string& filename, CObjReader &currObj);
};
// ObjReader.cpp
#include "ObjReader.h"

CObjReader::CObjReader(void):m_iNbofVertex(0){}

CObjReader::~CObjReader(void){}

void CObjReader::ReadObj(const std::string& filename, CObjReader &currObj)
{
	std::fstream fp;
	std::ofstream op(".\\output.txt");
	fp.open(filename.c_str(), std::ios::in);

	std::stringstream ss;
	std::string temp;
	float tempFloat = 0.0f;
	std::vector vertice;
	std::vector face;
	// Skip header
	do 
	{
		getline(fp, temp);
	} while (temp[0] == '#');

	// Process vertex and its normal
	while (temp[0] != '#')
	{
		ss << temp;
		ss >> temp;	// skip "vn"
		point3f tempPoint;
		ss >> tempPoint.normx >> tempPoint.normy >> tempPoint.normz;
		ss.clear();
		// get v
		getline(fp, temp);
		ss << temp;
		ss >> temp;	// skip 'v'
		ss >> tempPoint.x >> tempPoint.y >> tempPoint.z;
		ss.clear();
		vertice.push_back(tempPoint);
		// get vn for next point
		getline(fp, temp);
	}
	ss << temp;
	ss >> temp;	// skip '#'
	unsigned int a;
	ss >> a;
	ss.clear();
	assert(a == vertice.size());
	currObj.m_iNbofVertex = vertice.size();
	// Process face
	getline(fp, temp); // skip empty line
	getline(fp, temp); // Get first line of face data
	do 
	{
		ss.str("");	// Clean previous contents
		ss << temp;
		ss >> temp; // skip 'f'
		for (int i = 0; i < 3; i++)
		{
			ss >> temp; // temp: i//i
			temp.erase(temp.find('/'),temp.size()-temp.find('/'));
			face.push_back(std::stoi(temp));
		}
		ss.clear();
		getline(fp, temp);
	} while (temp[0] == 'f');
	std::cout << temp << std::endl;
	fp.close();
	
	// Save vertice and face info
	for (unsigned int i = 0; i < vertice.size(); i++)
	{
		op << "vn: ";
		op << vertice[i].normx << " " << vertice[i].normy << " " << vertice[i].normz << "\n";
		op << "v: ";
		op << vertice[i].x << " " << vertice[i].y << " " << vertice[i].z << "\n";
	}
	op << "# Face data\n";
	for (int i = 0; i < face.size();)
	{
		op << face[i] << " " << face[i+1] << " " << face[i+2] << "\n";
		i = i + 3;
	}
	op.close();
}

08 May 2013

Notes of VOSM shape model building framework

The information saved for fitting:

\ShapeModel\
ShapeModel.txt:
m_iNbOfSamples
m_iNbOfShapeDim
m_iNbOfPoints
m_iNbOfShapes
m_iNbOfEigenShapesAtMost
m_iNbOfShapeEigens
m_iNbOfEdges
m_iNbOfTriangles
m_fAverageShapeSize
m_fTruncatedPercent_Shape
m_PCAAlignedShapeMean.txt
m_PCAAlignedShapeEigenValues.txt
m_PCAAlignedShapeEigenVectors.txt
m_VOAlignedMeanShape.txt
m_VOReferenceShape.txt // Reference shape which is scaled back to the original size and translated to origin
m_vShapes.txt // All loaded shapes in a vector format
m_vAlignedShapes.txt // All aligned shapes in a vector format
m_vShape2DInfo.txt // shape information and face parts information
m_vEdge.txt
m_vTemplateTriangle2D.txt // Unnormalised triangles in the template shape
m_vNormalizedTriangle2D.txt // Normalised triangles in the template shape

.\Point2DDistributionModel\
Point2DDistributionModel.txt // number of points
m_VONormalizedEllipses.txt 

07 May 2013

VOSM 2D model building and fitting for face data

After successful compilation of VOSM we got "test_smbuilding.exe" and "test_smfitting.exe" which are responsible for model building and fitting respectively.
Preparation of database:
From the VOSM website we can download several database: link. For this post, I will use JIAPEI database. Within JIAPEI directory, there are two directories: .\annotations and .\images. The former one contains annotation of landmarks for both training and testing images. The later one contains the corresponding images. For each database there should be a ShapeInfo.txt which describes how the landmarks are connected as shape (with several parts).

Config of VOSM under Win7-32bit with VS2010 using CMake

VOSM is an open source 2D ASM/AAM C++ implementation which can be downloaded from here. Extract the file to a convenient directory, say .\vosm-0.3.3. If use CMake to compile the project we need to copy .\vosm-0.3.3\doc folder from other repository (say Linux) which contains a licience.txt that is required for CMake. We may need to modify the given CMakeLists.txt since the viriable OPENCV_INCLUDE_PATH was not set correctly. To solve this problem, change these lines (starting from line:248):
find_path(OPENCV_INCLUDE_PATH "cv.h"
        PATHS "/usr/include/opencv" "/usr/local/include/opencv"
        DOC "The path to OPENCV headers")
to:
 set(OPENCV_INCLUDE_PATH "DIRECTORY/TO/OPENCV/INCLUDE_FOLDER")
Then set both source and build directory in CMake to .\vosm-0.3.3 and Configure and Generate.
In .\vosm-0.3.3, open vosm.sln and Solution Explorer looks like:
For all projects whose names are all capital (such as ALL_BUILD), we don't need to configure them.
For all projects' name not start with (EXAMPLE) (such as smbuilding), change project (debug and release) property Configuration Type to Static library (.lib) and set Output Directory as .\vosm-0.3.3\lib\Debug\ or  .\vosm-0.3.3\lib\Release\. Set Target Extension to .lib. Here is a snapshot of property pages after this step:
For all projects except the one with capital name, configure OpenCV and Boost include and library directory (if they are not there) since they are VOSM's dependent libraries.
For all projects' name start with (EXAMPLE), configure correct OpenCV and Boost .lib file for each for both debug and release. The name of the .lib files are (OpenCV-2.4.4 Boost-1.4.9):

03 May 2013

A comparison of existing SSM/ASM/AAM implementation

After some researches, few popular libraries or source code for shape model implementations are found. They are:

  • Active Shape Model (ASM) and Active Appearance Model (AAM) in MATLAB (link): It is an relatively up-to-date implementation which provides both ASM and AAM using 2D and 3D model. The provided 2D sample is easy to understand and follow while the 3D sample is lack of description. It also has a script for selecting landmarks manually in 2D images. If applying user's own 3D shape, the 3D landmarks need to be prepared by user. The 2D sample can be run smoothly while the 3D case requires significant processing time during model training and template searching.
  • Multi-Resolution Active Shape Models in MATLAB (link): It has been written for many years and not conformable with new MATLAB.
  • Mikkel Stegman Active Appearance Model in C++ (link): This API is an open source C++ AAM implementation. However, the API was written based on MS VisionSDK which has been deprecated. The project itself also out-of-date as it only provides MSVC 6 and 2005 version.
  • Tim Cootes am_tools (link): It consists of a set of tools to build and play with AAMs. No source available.
  • Yao Wei asmlibrary (link): A library provides utilities for face model training and searching. It was written for human face related application. The source code is not available so it is not suitable for applying object other than face.
  • Active Shape Models with Stasm (link): It is a C++ library for locating landmarks of face and training and applying shape model in 2D case. Nevertheless, the author provides source code and good documentation for the library. The library should be able to be applied to other object in 3D case.