Disclosed are a method and a system for measuring a displacement of a structure based on fusion of asynchronous vision measurement data of natural target and acceleration data of the structure. A scale factor, which is a conversion factor between a pixel resolution of the image frame data and a distance resolution of real space, is calculated using image frame data taken at a predetermined first sample frequency of a stationary target outside the structure by a camera installed on the structure, and acceleration data measured at a predetermined second sample frequency by an accelerometer installed on the structure, over time. A vision-based displacement value of the structure is estimated using a feature point matching algorithm and the scale factor.