A video encoder including a processing block and an external memory storing a current frame and a reference frame. The processing block includes a memory interface, a local memory and a processor. The processor encodes the current frame in raster scan macroblock order for FMO using information from the reference frame, converts encoded information into compressed information, and organizes the compressed information according to a predetermined FMO. The processor organizes the compressed information according to any suitable FMO organization such as scattered, interleaved, etc. The processor stores the compressed information into multiple slice groups into the local memory or into the external memory, where the slice groups are organized according to the FMO. The processor loads a search window macroblock into the local memory if not already stored in the local memory. The processor may generate unfiltered reconstructed information and store the unfiltered reconstructed information into the local memory.