Systems and methods for improving operational efficiency of a video encoding system used to encode image data are provided. In embodiments, the video encoding system includes a low resolution pipeline that includes a low resolution motion estimation block, which generates downscaled image data by reducing resolution of the image data, performs a motion estimation search to determine motion vector candidates that are each indicative of location of a downscaled reference sample, and prunes one or more of the motion vector candidates. The video encoding system also includes a main pipeline that determines encoding parameters to be used to encode image data based at least partially the remaining motion vector candidates.