A computer implemented method of semantically categorizing a video stream through multimodal content classification, comprising dividing a designated video stream to a plurality of scenes by analyzing a visual content of a plurality of frames of the video stream to identify scene changes between consecutive scenes, applying a plurality of classification functions to each of a plurality of modalities extracted from each of the scenes to calculate a class probability for each of a plurality of known concepts detected in each scene, applying a plurality of multimodal classification functions on the class probability of the known concepts to calculate a scene category probability for each scene indicating a probability of the scene to be categorized in one or more semantic categories and categorizing the video stream to a stream category of the semantic categories by aggregating the category probability of the scenes.