A magnification system is described that provides a better user experience to users of desktop magnification, such as in conjunction with touch-based interface devices. The system includes an interactive panning mode that allows users to pan a magnified area of the desktop or application while still interacting with magnified elements, such as icons, files, and so forth. In the interactive panning mode, the user can pan the magnified desktop in a manner similar to traditional panning by selecting an area of the magnified desktop that does not contain user interface elements. The user can also click/touch buttons, UI elements, and interact with the magnified desktop in a normal fashion.