The present invention is directed to a system and method for interactive and iterative reconstruction in a manner that helps to reduce computational requirements by generating a model from a subset of the available data, and then refining that model using additional data. Example embodiments directed to scene reconstruction, reconstruct a 3D scene from a plurality of 2D images of that scene by first generating a model of the 3D scene from a subset a of the 2D images. The model can then be refined using specific characteristics of each image in the subset that are calculated using the other images in the subset. The model is further refined using images not in the original subset.