Techniques for performing continuous enrollment for face verification are provided. In one embodiment, a computing device can receive, from a user, an indication that the user wishes to authenticate himself/herself with the computing device via face verification. In response to the indication, the computing device can capture, using a camera, a series of images of the user's face and can authenticate the user by evaluating each of the series of images against a face template for the user, where the user is authenticated based on an N-th image in the series. Once the user has been authenticated, the computing device can select one or more images from the series prior to the N-th image and can add the selected images to the user's face template.