A semiconductor device includes a substrate, two gate structures, an interlayer dielectric layer and a material layer. The substrate has at least two device regions separated by at least one isolation structure disposed in the substrate. Each device region includes two doped regions disposed in the substrate. The gate structures are respectively disposed on the device regions. In each device region, the doped regions are respectively disposed at two opposite sides of the gate structure. The interlayer dielectric layer is disposed over the substrate and peripherally surrounds the gate structures. A top of the interlayer dielectric layer has at least one concave. The material layer fills the concave and has a top surface elevated at the same level with top surfaces of the gate structures. A ratio of a thickness of a thickest portion of the material layer to a pitch of the gate structures ranges from 1/30 to 1/80.