A semiconductor device includes a gate isolation structure on a shallow trench isolation (STI), a first epitaxial layer on one side of the gate isolation structure, a second epitaxial layer on another side of the gate isolation structure, first fin-shaped structures directly under the first epitaxial layer, and second fin-shaped structures directly under the second epitaxial layer, in which the STI surrounds the first fin-shaped structures and the second fin-shaped structures.