Video data matching includes both a single region of data and sequences of regions of data. In an embodiment, a video processing system selects, from a first matrix row corresponding to a test appearance model, one or more other appearance models as a function of similarity measures populating the first matrix row. After selection of the one or more other appearance models, the system then selects, from other matrix rows corresponding to the one or more other appearance models selected in the first step, one or more additional appearance models as a function of the similarity measures populating the other matrix rows. The system then ranks the appearance models selected from the first matrix row and the other matrix rows.