A multi-color image processing apparatus comprises an image pickup part (101) and a signal processing part (104). The image pickup part (101) comprises a color separating part (10) that separates visible light into, at least, light of first color components and light of second color components different from the first color components; and first and second image pickup elements (12, 14) that receive the first and second color component lights, respectively. The image pickup part (101) reads, onto the pixel matrix of the first image pickup element (12), and acquires, by thinning out the pixels, images of the first color components on field-by-field basis, while acquiring images of the second color components on the pixel matrix of the second image pickup element (14) on field-by-field basis. The signal processing part (104) comprises a motion estimating part (107) that estimates, based on the second color component images, a motion in a moving image, which is constituted by the second color component images, to output motion information; an image generating part (108) that generates, based on the first color component images and the motion information, combined images of the first color components in the respective fields and outputs them as first combined images having a higher spatial resolution than the first color component images; and a multi-color image combining part (109) that generates and outputs, based on the first combined images and the second color component images, multi-color moving images including the first and second color components.