The invention belongs to the technical field of computer image processing, and particularly discloses a garment replacement-based color matching assessment method. The method comprises the steps of performing foreground detection and midline detection according to a daily user picture and a selected and purchased garment; performing human face detection on the user picture, and simulating garmentreplacement; and extracting picture color features after the garment replacement, performing clustering, and assessing a garment color and skin matching effect (color harmony degree) by utilizing a Matsuda color harmony degree model. The invention provides the foreground detection and midline detection-based garment replacement method; and the skin color and garment color matching after the garment replacement can be subjected to effective aesthetic assessment.