Systems and methods for detecting demographic bias in automatic speech recognition (ASR) systems. Corpuses of transcriptions from different demographic groups are analyzed, where one of the groups is known to be susceptible to bias and another group is known not to be susceptible to bias. A difference between the transcription accuracy for the first group and a transcription accuracy for a second group is measured. ASR accuracy for each group is measured and compared to each other using both statistics-based and practicality-based methodologies to determine whether a given ASR system or model exhibits a meaningful level of bias. Based on the statistical significance and the practical significance, an alert including a recommendation to adjust the ASR model is generated.