A semiconductor device includes a gate structure on a substrate and an epitaxial layer adjacent to the gate structure, in which the epitaxial layer includes a first buffer layer, an anisotropic layer on the first buffer layer, a second buffer layer on the first buffer layer, and a bulk layer on the anisotropic layer. Preferably, a concentration of boron in the bulk layer is less than a concentration of boron in the anisotropic layer, a concentration of boron in the first buffer layer is less than a concentration of boron in the second buffer layer, and the concentration of boron in the second buffer layer is less than the concentration of boron in the anisotropic layer.