Systems and methods for event and object detection and annotation in the video streams may include extracting a plurality of features in a picture in a video frame, grouping at least a portion of the plurality of features into at least one object, determining a region for the at least one object, assigning object identifiers to the at least one object and encoding the object identifiers into the bitstream. Feature-based rate distortion optimization may be employed for video coding including extracting a set of features from a picture in the video, generating a relevance map for the extracted features, determining a relevance score for portions of the picture using the relevance map, and encoding the portion of the picture with a bit rate determined at least in part by the relevance score.