After forming a gate spacer on each sidewall of a sacrificial gate structure, portions of each dielectric fin cap portion underneath the gate spacer is intentionally etched and undercut regions that are formed are filled and pinched off with a dielectric material of a conformal dielectric liner. Portions of the conformal dielectric liner in the undercut regions are not subject to the undercut during an epitaxial pre-clean process performed prior to forming an epitaxial source region and an epitaxial drain region on opposite sides of the sacrificial gate structure and remain in the undercut regions after forming the epitaxial source region and the epitaxial drain region.