Systems, apparatuses, and methods for implementing cache line re-reference interval prediction using a physical page address are disclosed. When a cache line is accessed, a controller retrieves a re-reference interval counter value associated with the line. If the counter is less than a first threshold, then the address of the cache line is stored in a small re-use page buffer. If the counter is greater than a second threshold, then the address is stored in a large re-use page buffer. When a new cache line is inserted in the cache, if its address is stored in the small re-use page buffer, then the controller assigns a high priority to the line to cause it to remain in the cache to be re-used. If a match is found in the large re-use page buffer, then the controller assigns a low priority to the line to bias it towards eviction.