A method includes forming a transistor, which includes forming a dummy gate stack over a semiconductor region, and forming an Inter-Layer Dielectric (ILD). The dummy gate stack is in the ILD, and the ILD covers a source/drain region in the semiconductor region. The method further includes removing the dummy gate stack to form a trench in the first ILD, forming a low-k gate spacer in the trench, forming a replacement gate dielectric extending into the trench, forming a metal layer to fill the trench, and performing a planarization to remove excess portions of the replacement gate dielectric and the metal layer to form a gate dielectric and a metal gate, respectively. A source region and a drain region are then formed on opposite sides of the metal gate.