An image processing apparatus for obtaining an output image by performing a filter processing of an input image, includes a setting unit configured to set a plurality of local regions in the input image, a deriving unit configured to derive a plurality of coefficients corresponding to each of the plurality of local regions based on statistics indicating variations in pixel values in the plurality of local regions, and a filter processing unit configured to obtain an output pixel value by performing a liner transformation of the input image based on the plurality of the coefficients, wherein the plurality of local regions include a region that is partitioned diagonally to a raster scan direction; wherein the filter processing unit performs the liner transformation of the input image based on a coefficient adaptively selected from the plurality of coefficients.