A method includes forming isolation regions extending from a top surface of a semiconductor substrate into the semiconductor substrate, and forming a hard mask strip over the isolation regions and a semiconductor strip. The semiconductor strip is between two neighboring ones of the isolation regions. A dummy gate strip is formed over the hard mask strip. A lengthwise direction of the dummy gate strip is perpendicular to a lengthwise direction of the semiconductor strip, and a portion of the dummy gate strip is aligned to a portion of the semiconductor strip. The method further includes removing the dummy gate strip, removing the hard mask strip, and recessing first portions of the isolation regions that are overlapped by the removed hard mask strip. A portion of the semiconductor strip between and contacting the removed first portions of the isolation regions forms a semiconductor fin.