Systems and methods for managing datasets produced by alert-triggering search queries in data aggregation and analysis systems. An example method may comprise: executing, by one or more processing devices, a search query on a portion of searchable data associated with a time window to produce a dataset comprising one or more results; responsive to determining that at least a portion of the dataset satisfies a triggering condition defining an alert associated with the search query, generating an instance of the alert; associating, by a memory data structure, the instance of the alert with an identifier of the search query and a time parameter specifying the time window; receiving, from a client computing device, a request for the portion of the dataset; and responsive to determining that the portion of the dataset is not stored in the memory in a manner associating it with the instance of the alert, reproducing the portion of the dataset by re-executing the search query in view of the time parameter.