This disclosure describes video encoding techniques capable of reducing the number of processing cycles and memory transfers necessary to encode a video sequence. In this manner, the disclosed video encoding techniques may increase video encoding speed and reduce power consumption. In general, the video encoding techniques make use of a candidate memory that stores video blocks in columns corresponding to a search space for a motion estimation routine. A memory control unit addresses the candidate memory to retrieve multiple pixels in parallel for simultaneous comparison to pixels in a video block to be encoded, e.g., using Sum of Absolute Difference (SAD) or Sum of Squared Difference (SSD) techniques. A difference processor performs the parallel calculations. In addition, for subsequent video blocks to be encoded, the candidate memory can be incrementally updated by loading a new column of video blocks, rather than reloading the entire search space.