Methods, systems, and apparatus, including computer programs encoded on computer storage media, for automatically classifying static analysis rules as being anomalous or not. One of the methods includes receiving alerts generated by a particular static analysis rule for a plurality of different software projects analyzed by a static analysis system. For each project, a respective alert proportion metric value is computed. Each of the plurality of different software projects is classified according to the alert proportion metric values as being one non-outlier projects or outlier projects. If more than a threshold number of projects were classified as being outlier projects for the particular static analysis rule, the particular static analysis rule is classified as an anomalous static analysis rule.