A method of forming a semiconductor device is provided. At least one gate structure including a dummy gate is formed on a substrate. A contact etch stop layer and a dielectric layer are formed to cover the gate structure. A portion of the contact etch stop layer and a portion of the dielectric layer are removed to expose the top of the gate structure. A dry etching process is performed to remove a portion of the dummy gate of the gate structure. A hydrogenation treatment is performed to the surface of the remaining dummy gate. A wet etching process is performed to remove the remaining dummy gate and thereby form a gate trench.