Methods, systems, and apparatus, including computer program products, for identifying images relevant to a query are disclosed. An image search subsystem selects images to reference in image search results that are responsive to a query based on an image relevance model that is trained for the query. An independent image relevance model is trained for each unique query that is identified by the image search subsystem. The image relevance models can be applied to images to order image search results obtained for the query. Each relevance model is trained based on content feature values of images that are identified as being relevant to the query (e.g., frequently selected from the image search results) and images that are identified as being relevant to another unique query. The trained model is applied to the content feature values of all known images to generate an image relevance score that can be used to order search results for the query.