The present invention discloses a device for processing reference frame in image, wherein the device comprises a control circuit (301), an address pointer register (302), an on-position register (303), a comparator (304) and a reference frame list memory unit (305). The invention also discloses a method for processing reference frame in image and a decoder thereof. According to the plan of the invention, the rearranging of reference frame is realized through hardware. The processing speed can be greatly increased compared with the prior art.