A vibration damping system for a charged particle beam apparatus according to the present invention includes a column through which a charged particle beam passes, a vibration detection unit that detects vibration of the column, a damping mechanism that applies vibration to the column to suppress the vibration of the column, and a control device that controls the damping mechanism. The control device includes a damping gain control unit that amplifies a detection signal of the vibration detection unit with a set amplification factor and outputs an amplified detection signal as a control signal to the damping mechanism, and a saturation suppression unit that adjusts a feedback gain value of the damping gain control unit according to a detection signal of the vibration detection unit, a signal of the damping mechanism, and a maximum output value and a minimum output value of the damping mechanism.