A video signal processing device includes first access circuitry for writing a video signal in a memory capable of storing a video signal of at least two picture frames, and second access circuitry for reading the video signal from the memory. Comparision circuitry then compares a first address accessed by the first access circuitry with a second address accessed by the second access circuitry. The second address is then shifted by the amount of an integer of picture frames in accordance with the result of the comparison either at the moment when the first access circuitry is granted access to the memory or when access of the first access circuitry to the memory is stopped.