A system and method for capturing and processing aerial images of structures to assess structural damage is disclosed. The system comprises an aerial system associated with an aerial vehicle, which is in contact with a ground system. The aerial system is used to capture images of selected locations associated with known structures and crop those images around the selected locations. The cropped images are then fed into a damage classifier that automatically classifies the degree of damage done to structures at the selected locations. The system and method may be deployed to quickly assess damage of structures in a disaster area.