A display controller for displaying multiple windows and associated memory access method are provided. The display controller receives a first video source and a second video source for displaying multiple windows, and includes a line buffer, a deinterlacer, a scaler, and a memory interface unit. The line buffer buffers pixel data of a non-overlapped area of a main image associated with the first video source, and pixel data of a sub image associated with the second video source. The deinterlacer is coupled to the line buffer for selectively deinterlacing data in the line buffer. The scaler is coupled to the deinterlacer for selectively scaling data outputted from the deinterlacer. The memory interface unit is coupled to the line buffer for accessing an external memory.