A real-time video exploration (RVE) system that allows users to pause a pre-recorded video (e.g., a movie) and change viewing positions and angles of one or more scenes. The RVE system may also allow users to select, manipulate, and modify objects within one or more scenes. The RVE system may render new video of the scenes and stream the new video to the user's client device. Using the RVE system, a user may create modified versions of one or more scenes to replace original scenes in a video to thus create a modified version of the video. The RVE system may allow the user to record the modified version of the video, and may allow the user to share the modified version of the video with other viewers.