Described is a system for ordering images. The system receives a plurality of images. Image features are extracted from each image. A set of all possible image pairs are generated for all images. A similarity metric with weights is generated between the images in each image pair in the set, with a net similarity metric thereafter generated by combining the similarity metrics. The images are then ordered according to the net similarity metrics to generate a computer-ordered set of images. The computer-ordered set of images is then displayed to the user, which allows the user to re-order the images to generate a user-ordered set of images. The weights are then optimized to minimize the distance between the computer-ordered set of images and the user-ordered set of images. The similarity metrics are then re-weighted, with the images thereafter being re-ordered according to the new metrics.