Systems and methods for driver behavior recognition is provided. In one embodiment a computer implemented method includes receiving image data associated with a general objects. The method also includes identifying a reactive object and an inert object from the general objects based on the image data. An ego reactive graph is generated for the reactive object based on a reactive feature of the reactive object and a reactive position vector. An ego inert graph is generated for the inert object based on an inert feature of the inert object and an inert distance. The method further includes performing interaction modeling based on the ego reactive graphs and the ego inert graphs to generate updated features. The method also includes performing temporal modeling on the updated features. The method further includes determining an egocentric representation of a tactical driver behavior based at least in part on the updated features.