Innovations in flexible reference picture management are described. For example, a video encoder and video decoder use a global reference picture set (“GRPS”) of reference pictures that remain in memory, and hence are available for use in video encoding/decoding, longer than conventional reference pictures. In particular, reference pictures of the GRPS remain available across random access boundaries. Or, as another example, a video encoder and video decoder clip a reference picture so that useful regions of the reference picture are retained in memory, while unhelpful or redundant regions of the reference picture are discarded. Reference picture clipping can reduce the amount of memory needed to store reference pictures or improve the utilization of available memory by providing better options for motion compensation. Or, as still another example, a video encoder and video decoder filter a reference picture to remove random noise (e.g., capture noise due to camera imperfections during capture).