Methods, systems, and apparatus, including computer programs encoded on computer storage media, for updating a trained gesture recognition model deployed on a neuromorphic processor that has been trained to process data that characterizes the new gesture and to determine a gesture classification for the gesture are described. A method includes receiving data that characterizes a new gesture and processing the data to generate a new embedding in a latent space. For each of multiple clusters of reference embeddings in the latent space, a respective distance in the latent space between the cluster of reference embedding and the new embedding is determined. A determination is made, based on applying one or more learning rules to the distances, one or more procedures to update the gesture recognition model. A determination is made, in accordance with the determined procedure(s), an update to values of one or more parameters of the gesture recognition model.