An interlayer insulating film is formed. Then a first gate electrode and a second gate electrode are buried in the interlayer insulating film. Then, an anti-diffusion film is formed over the interlayer insulating film, over the first gate electrode, and over the second gate electrode. Then, a first semiconductor layer is formed over the anti-diffusion film which is present over the first gate electrode. Then, an insulating cover film is formed over the upper surface and on the lateral side of the first semiconductor layer and over the anti-diffusion film. Then, a semiconductor film is formed over the insulating cover film. Then, the semiconductor film is removed selectively to leave a portion positioned over the second gate electrode, thereby forming a second semiconductor layer.