A fleet management system implements a rewards-based feedback system to receive feedback from users of a rideshare service that provides rides using autonomous vehicles (AVs). The fleet management system maintains user accounts associated with each user and provides reward points to each user account. When the user is riding in an AV, the user can access a user interface to select a portion of the reward points and reward them to the AV. The fleet management system may analyze the point rewards to identify individual user preferences, to identify preferences across users, or to identify AVs for maintenance or other types of modification.