A method includes removing a first dummy gate stack and a second dummy gate stack to form a first trench and a second trench. The first dummy gate stack and the second dummy gate stack are in a first device region and a second device region, respectively. The method further includes depositing a first gate dielectric layer and a second gate dielectric layer extending into the first trench and the second trench, respectively, forming a fluorine-containing layer comprising a first portion over the first gate dielectric layer, and a second portion over the second gate dielectric layer, removing the second portion, performing an annealing process to diffuse fluorine in the first portion into the first gate dielectric layer, and at a time after the annealing process, forming a first work-function layer and a second work-function layer over the first gate dielectric layer and the second gate dielectric layer, respectively.