Techniques for masking images based on a particular task are described. A system masks portions of an image that are not relevant to a particular task, thus, reducing the amount of data used by applications for image processing tasks. For example, images to be processed using a hair color classification model are masked so that only portions that show the person's hair are available for the model to analyze. The system configures different masker components to mask images for different tasks. A masker component can be implemented at a user device to mask images prior to sending to an application/task-specific model.