A method of manufacturing a semiconductor device having a silicon substrate containing an impurity diffusion layer is disclosed, that comprises the steps of doping impurities to the silicon substrate through a silicon oxide film with a thickness of 2.5 nm or less at an accelerating voltage of 3 keV or less, the silicon oxide film being formed on the silicon substrate and annealing the silicon substrate with the oxide film left.