An evacuation structure of a charged particle beam device includes: a vacuum chamber provided with a charged particle source; vacuum piping connected to the vacuum chamber; a main vacuum pump which is connected via the vacuum piping and evacuates the inside of the vacuum chamber; a non-evaporable getter pump disposed at a position between the vacuum chamber and the main vacuum pump in the vacuum piping; and a coarse evacuation port connected at a position between the vacuum chamber and the non-evaporable getter pump in the vacuum piping The coarse evacuation port includes: a coarse evacuation valve that opens and closes the coarse evacuation port; and a leak valve to open the vacuum chamber to the atmosphere.