The observation system images a specimen under illumination lights applied at different timings through irradiation windows arranged near one image sensor, generates optical information included in each image, calculates values of light volumes included in the optical information output from identical-point originated pixels, which have imaged an identical point as the same characteristic point on the specimen, in the respective pixels as a relative distance from an image sensor, compares the optical information between the pixels, and determines the identical point on the specimen as a flat surface portion or an inclined portion on the basis of a comparison result, associates the distance with the flat surface portion or the inclined portion, estimates a surface shape of the specimen, and displays a three-dimensional image of the specimen.