Systems and methods are described for identifying at least one sports team depicted in media content, such as image or video data. Features of the media content may be provided as input to a first set of classification models that are each trained to identify at least one type of scene associated with one or more sports, then features of the media content may be provided to a second set of classification models trained to identify at least one object associated with one or more sports. Once a sport depicted in the media content is determined based on the first and second set of classification models, the system may determine a team depicted in the media content based at least in part by comparing aspects of the media content to stored data associated with various teams that play the identified sport.