Provided herein are systems and methods for implementing a video surveillance system. In one or more examples, the improved system can include a video managing system that applies one or more watchlists, to video footage captured by surveillance cameras to apply one or more processing policies indicated by the one or more watchlists. In some embodiments, a watchlist may indicate a plurality of events of interest (e.g., specific classes of objects, a presence of one or more persons, or one or more persons of interest, etc.) that should be detected from video footage. Through these mechanisms, video processing can be increased since the types of events of interest that need to be detected within the video footage are reduced. Non-transitory storage requirements are reduced, and human labor requirements to monitor, analyze, and review video surveillance footage is reduced.