High voltage semiconductor device and manufacturing method thereof are disclosed. The high voltage semiconductor device includes a semiconductor substrate, a gate structure, at least one first isolation structure and at least one second isolation structure, and at least one first drift region. The gate structure is disposed on the semiconductor substrate. The first isolation structure and the second isolation structure are disposed in an active area of the semiconductor substrate at a side of the gate structure. An end of the second isolation structure is disposed between the first isolation structure and the gate structure, and an end of the first isolation structure is disposed between the first doped region and the second isolation structure. A bottom of the at least one first isolation structure and a bottom of the at least one second isolation structure are deeper than a bottom of the first drift region.