The present invention discloses an image distortion correction method of a scanning electron microscope and belongs to the computer vision field. The method comprises the following steps that: S1, images are acquired continuously on a preset time sequence, and a relationship Dd (t) between time shift distortion Dd and image acquisition time t is established; S2, drift distortion is removed in advance according to Dd (t) obtained in the S1, and a standard target is photographed, so that sparse image pixel position information and a corresponding distortion vector sample set are obtained; S3, arelation between the pixel coordinates u of any one pixel in an image plane and an inherent distortion vector l corresponding to the pixel is established, so that the inherent distortion model Df of imaging is obtained through calculation; and S4, the distortion model matrix Df is solved by using the sparse position information and the corresponding distortion vector sample set which are obtainedin the S2, and Df (u) and Dd (t) are used in combination to perform distortion correction on the images. According to the method of the invention, distortion modeling and correction are performed on two main distortion fields of the images of the scanning electron microscope. The method can be applied to practical engineering.