A method and apparatus are provided for determining one or more object models used by an autonomous vehicle to predict the behavior of detected objects. The autonomous vehicle may collect and record object behavior using one or more sensors. The autonomous vehicle may then communicate the recorded object behavior to a server operative to determine the object models. The server may determine the object models according to a given object classification, a particular driving environment, or a combination thereof. The server may then communicate the object models to the autonomous vehicle for use in predicting the actions of detected objects and for responding accordingly.