An active appearance model is built by arranging the training images in its training library into a hierarchical tree with the training images at each parent node being divided into two child nodes according to similarities in characteristic features. The number of node levels is such that the number of training images associated with each leaf node is smaller than a predefined maximum. A separate AAM, one per leaf node, is constructed using each leaf node's corresponding training images. In operation, starting at the root node, a test image is compared with each parent node's two child nodes and follows a node-path of model images that most closely matches the test image. The test image is submitted to an AAM selected for being associated with the leaf node at which the test image rests. The selected AAM's output aligned image may be resubmitted to the hierarchical tree if sufficient alignment is not achieved.