The model generation system may generate texture maps for a texture based on a material image. A material image is an image of a physical material that describes the color (e.g., red-green-blue (RGB) color system model) of the physical material. The model generation system may identify a material class for the physical material depicted in the material image by applying a machine-learning model to the material image. The model generation system may then identify a texture map model that generates texture maps for the physical material based on the material image. The texture map model is a machine-learning model that is trained to generate texture maps for material images of a particular material class. The texture maps generated by the texture map model may include texture maps of standard texture values, such as metalness and roughness.