What is disclosed is system and method for contemporaneously reconstructing images of a scene illuminated with unstructured and structured illumination sources. In one embodiment, the system comprises capturing a first 2D image containing energy reflected from a scene being illuminated by a structured illumination source and a second 2D image containing energy reflected from the scene being illuminated by an unstructured illumination source. A controller effectuates a manipulation of the structured and unstructured illumination sources during capture of the video. A processor is configured to execute machine readable program instructions enabling the controller to manipulate the illumination sources, and for effectuating the contemporaneous reconstruction of a 2D intensity map of the scene using the second 2D image and of a 3D surface map of the scene using the first 2D image. The reconstruction is effectuated by manipulating the illumination sources.