A risk maneuver assessment system and method to generate a perception of an environment of a vehicle and a behavior decision making model for the vehicle; a sensor system configured to provide the sensor input in the environment for filtering target objects; one or more modules configured to map and track target objects to make a candidate detection from multiple candidate detections of a true candidate detection as the tracked target object; apply a Markov Random Field (MRF) algorithm for recognizing a current situation of the vehicle and predict a risk of executing a planned vehicle maneuver at the true detection of the dynamically tracked target; apply mapping functions to sensed data of the environment for configuring a machine learning model of decision making behavior of the vehicle; and apply adaptive threshold to cells of an occupancy grid for representing an area of tracking of objects within the vehicle environment.