A method of manufacturing a semiconductor device includes: forming a stack of semiconductor layers and sacrificial layers alternately arranged over a substrate; patterning the stack to form a stacking structure on the substrate; disposing a sacrificial gate structure on the substrate, where the sacrificial gate structure covers a portion of the stacking structure; removing portions of the stacking structure not overlapped with the sacrificial gate structure; disposing source/drain regions at opposite sides of the sacrificial gate structure, where the semiconductor layers in the remained stacking structure connect between the source/drain regions; removing the sacrificial gate structure and rest of the sacrificial layers to form a cavity accessibly revealing the semiconductor layers; forming a semiconductor material to cover the semiconductor layers; performing a thermal process to transfer the semiconductor material into a Si-containing layer and a Ge-containing layer, where the Si-containing layer is disposed over the semiconductor layers, and the Ge-containing layer is interposed between the Si-containing layer and the semiconductor layers; and forming a gate structure in the cavity and over the remained stacking structure.