A method and a system for summarizing a concept are provided. A query corresponding to a concept is received from a user. A plurality of images and corresponding descriptive information may be collected based on the query. The plurality of images and the descriptive information may be processed to form feature vectors and processed descriptive information respectively. Further, one or more topics may be identified for the plurality of images. Each of the plurality of images may be assigned with one or more topic distribution values corresponding to the one or more topics. The one or more topics correspond to the processed descriptive information. A sparse set of images may be determined based on the feature vectors and the assigned topic distribution values, to summarize the concept. Also, a target summary may be built from the summarized concept, by regularizing one or more distribution constraints.