A prediction set determining section selects a prediction set from a prediction set group including a plurality of prediction sets having different combinations of prediction modes corresponding to different prediction directions. Further, a prediction mode determining section selects a prediction mode from the prediction set thus selected. An entropy encoding section encodes the prediction set thus selected, the prediction mode thus selected, and residual data between an input image and a predicted image formed on the basis of the prediction set and the prediction mode. This allows an image encoding device to carry out predictions from more various angles, thereby improving prediction efficiency.