A method includes forming a gate stack including a gate electrode on a first semiconductor fin. The gate electrode includes a portion over and aligned to a middle portion of the first semiconductor fin. A second semiconductor fin is on a side of the gate electrode, and does not extend to under the gate electrode. The first and the second semiconductor fins are spaced apart from, and parallel to, each other. An end portion of the first semiconductor fin and the second semiconductor fin are etched. An epitaxy is performed to form an epitaxy region, which includes a first portion extending into a first space left by the etched first end portion of the first semiconductor fin, and a second portion extending into a second space left by the etched second semiconductor fin. A first source/drain region is formed in the epitaxy region.