Exemplary embodiments relate to the automatic generation of captions for visual media in the form of a consistent story or narrative. According to some embodiments, story generation may be applied to a live video. As a user records live video, a system may analyze metadata, the frames of the video, and/or the audio to extract context information. The system may integrate this information with information from the user's social network and a personalized language model built using public-facing language from the user. The system may generate multiple captions for the video, where subsequent captions are based at least partially on previous captions. Captions may be generated in a story format so as to be consistent with each other. Information that is inconsistent with the story may be excluded from the captions unless contextual factors indicate that the story should change subject.