A system and a method are disclosed for calibrating an image acquisition system of a farming machine. The farming machine captures images of objects in an environment as the farming machine moves through an environment. Based on the captured images, a represented three-dimensional (3D) model of the environment representing the objects in a three-dimensional space is generated. From the represented 3D model, a represented location of an object (e.g., building, lamppost, tree) is determined. The represented location of the object is compared to a reference location of the object and the comparison is used to calibrate the image acquisition system by modifying positions, orientations, or optical settings of one or more cameras in the image acquisition system.