Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for receiving a first input associated with a bookmark and, in response to receiving the first input, obtaining context data for deriving annotations for the bookmark. An annotation engine of a system obtains multiple annotations that are derived using information included in the context data and a transcription of the first input associated with the bookmark The annotation engine generates an importance value for each annotation. The importance value characterizes a relationship between each annotation and at least one of: the information included in the context data; or the transcription of the first input. The annotation engine generates a set of annotations for retrieving the bookmark. Each annotation in the set of annotations is generated from the obtained multiple annotations and each annotation has an importance value that exceeds a threshold value.