According to one aspect, keyframe based autonomous vehicle operation may include collecting vehicle state information and collecting environment state information. A size of an object within the environment, a distance between the object and the autonomous vehicle, and a lane structure of the environment through which the autonomous vehicle is travelling may be determined. A matching keyframe model may be selected based on the size of the object, the distance from the object to the autonomous vehicle, the lane structure of the environment, and the vehicle state information. Suggested limits for a driving parameter associated with autonomous vehicle operation may be generated based on the selected keyframe model. The autonomous vehicle may be commanded to operate autonomously according to the suggested limits for the driving parameter.