A semiconductor device includes a semiconductor substrate, a radiation-sensing region, at least one isolation structure, and a doped passivation layer. The radiation-sensing region is present in the semiconductor substrate. The isolation structure is present in the semiconductor substrate and adjacent to the radiation-sensing region. The doped passivation layer at least partially surrounds the isolation structure in a substantially conformal manner.