After removal of the disposable gate structures to form gate cavities in a planarization dielectric layer, a silicon oxide layer is conformally deposited on silicon-oxide-based gate dielectric portions in the gate cavities. A portion of the silicon oxide layer can be nitridated to form a silicon oxynitride layer. A patterned masking material layer can be employed to physically expose a semiconductor surface from a first-type gate cavity. The silicon oxide layer can be removed while preserving an underlying silicon-oxide-based gate dielectric portion in a second-type gate cavity. A stack of a silicon oxynitride layer and an underlying silicon-oxide-based gate dielectric can be protected by a patterned masking material layer in a third-type gate cavity during removal of the silicon oxide layer in the second-type gate cavity. A high dielectric constant gate dielectric layer can be formed in the gate cavities to provide gate dielectrics of different types.