Blocking dielectric structures and/or thicker barrier metal films for preventing or reducing fluorine diffusion are provided. A blocking dielectric layer can be formed as an outer layer of a memory film in a memory stack structure extending through electrically insulating layers and sacrificial material layers. After formation of backside recesses by removal of the sacrificial material layers, dopants can be introduced into physically exposed portions of the blocking dielectric layer, for example, by plasma treatment or thermal treatment, to form silicon oxynitride regions which can reduce or prevent fluorine diffusion. Alternatively or additionally, a set of metal oxide blocking dielectric material portions can be formed in the backside recesses to retard or prevent fluorine diffusion. To minimize adverse impact on the electrically conductive layers formed in the backside recesses, the blocking dielectric material portions can be laterally recessed from a trench employed to form the backside recesses.