A solution is provided for selectively caching streaming media based on playback data associated with the streaming media. Each media stream is divided into a number of media chunks. Spatial locality and temporal locality of each media chunk is analyzed based on playback data of the media chunk and the corresponding media stream. Based on the spatial locality and temporal locality, a spatial score and temporal score are generated, respectively. Aggregation of the spatial score and temporal score produces a final score for the media chunk. The media chunks are ranked based on their final scores, where all or a number of media chunks are selected based on their rankings for being cached. The ranking of a cached media chunk decays over time, and the ranking of a cached media chunk is dynamically recalculated when another video chunk is to be cached.