A fast motion estimation system that determines a reference macroblock or sub-block combination within a reference frame for a current macroblock in a current frame includes a memory, a reference macroblock search circuit, a sub-block combination search circuit, and a comparator circuit. The reference macroblock search circuit determines a motion vector, multiple difference values, and a cost value for each macroblock within the reference frame according to a fast motion estimation search pattern, and stores the motion vector and the difference values in the memory. The sub-block combination search circuit searches the motion vector and the difference values in the memory for determining a corresponding one of multiple lowest cost sub-block combinations for each of multiple sub-block motion modes. The comparator circuit determines a lowest cost macroblock and selects from among the lowest cost macroblock and the lowest cost sub-block combinations to determine the reference macroblock.