An image processing apparatus includes: an orthogonal transforming section which performs an orthogonal transform for image data to generate a transform coefficient; and a quantization factor detecting section which detects a quantization factor used in a previous encoding process, using the transform coefficient, wherein the quantization factor detecting section independently performs a process of detecting the quantization factor from the transform coefficient generated by orthogonally transforming a luminance component of the image data and a process of detecting the quantization factor from the transform coefficient generated by orthogonally transforming a color difference component of the image data.