A system and method for utilizing a temporal recurrent network for online action detection that include receiving image data that is based on at least one image captured by a vehicle camera system. The system and method also include analyzing the image data to determine a plurality of image frames and outputting at least one goal-oriented action as determined during a current image frame. The system and method further include controlling a vehicle to be autonomously driven based on a naturalistic driving behavior data set that includes the at least one goal-oriented action.