An online system extracts features from an application linked to the online system. The application is used by users of the online system and posts content to the online system. A trained model is applied to the extracted features to generate a quality score for the application. The trained model is trained using features extracted from a set of training applications and quality scores manually assigned to the training applications, wherein the manually assigned quality scores indicate whether each training application satisfies a set of criteria and the generated quality score represents a probability of the application satisfying the set of criteria. Based on the quality score, content provided by the application is ranked for presentation to a user of the online system in relation to other content of the online system. The online system presents the content provided by the application to the user according to the ranking.