A method includes etching a portion of a semiconductor material between isolation regions to form a trench, forming a semiconductor seed layer extending on a bottom surface and sidewalls of the trench, etching-back the first semiconductor seed layer until a top surface of the semiconductor seed layer is lower than top surfaces of the isolation regions, performing a selective epitaxy to grow a semiconductor region from the semiconductor seed layer, and forming an additional semiconductor region over the semiconductor region to fill the trench.