The disclosed systems and methods provide a novel framework that provides mechanisms for performing cost-effective, accurate and scalable detection and recognition of fine-grained events. The framework functions by training high precision and high recall object/optical character recognition (OCR) models and aligning video frames to text commentaries of the videos (e.g., licensed play-by-play). The disclosed framework operates as a single algorithm that performs multimodal alignments between events/actions within videos and their prescribed text. Thus, the disclosed framework is able to scale to fine-grained action categories across different venues by delving into the key frames and key aspects of a video to identify particular actions performed by particular actors, thereby providing the novelty of fine-granted action detection and recognition.