The invention provides a anti-aliasing image scaler and a anti-aliasing method thereof. The anti-aliasing image scaler includes an input, an output, a scaler cross bus, a horizontal scaler, a verticalscaler, and an input/output buffer. The vertical scaler or the horizontal scaler includes a anti-aliasing module. The anti-aliasing module includes a tap unit, a median filtering unit, a scoring unitand a mixing unit. The tap unit includes a tap for storing a pixel value of an image point to be processed currently; and the tap is shared by the median filtering unit and the scoring unit. The median filtering unit is used for calculating horizontal median filtering results and vertical median filtering results. The scoring unit is used to calculate horizontal and vertical scores, respectively,and obtain a final score. Both the horizontal and vertical scores include a first score and a second score, and the horizontal and vertical scores are used to determine whether it is a near horizontal edge or a near vertical edge, respectively. The mixing unit mixes the horizontal score, the horizontal median filtering results or the vertical score, the vertical median filtering result, and multi-phase scaler results with each other according to the final score to obtain a final anti-aliasing result.