A semiconductor device includes a substrate, a gate structure, a sidewall spacer, and an epitaxial layer. The gate structure is disposed on the substrate, and the substrate has at least one recess disposed adjacent to the gate structure. The sidewall spacer is disposed on at least two sides of the gate structure. The sidewall spacer includes a first spacer layer and a second spacer layer, and the first spacer layer is disposed between the gate structure and the second spacer layer. The epitaxial layer is disposed in the recess, and the recess is a circular shaped recess. A distance between an upmost part of the recess and the gate structure is less than a width of the sidewall spacer.