Provided is a semiconductor device including a substrate, an isolation structure, a barrier structure, a first conductive layer, a second conductive layer, a first gate dielectric layer, and a second gate dielectric layer. The substrate has a first region and a second region. The barrier structure is located on the isolation structure. The first conductive layer is located on the first region. The second conductive layer is located on the second region. The first gate dielectric layer is located between the first conductive layer and the substrate in the first region. The second gate dielectric layer is located between the second conductive layer and the substrate in the second region. The first gate dielectric layer and the second gate dielectric layer are separated by the isolation structure. A method of manufacturing the semiconductor device is also provided.