A computer-implemented method for identifying relevant subgroups, which are relevant for training a subgroup-robust classifier, in a training dataset associated with a machine learning model includes receiving a classification dataset wherein subgroups are unlabeled. For each data point in the classification dataset, the method uses gradient space partitioning (GraSP) to identify a gradient representation of each data point by extracting an associated gradient of a logistic regression classification loss with respect to weights of a logistic regression. The gradient representations are clustered to provide estimated subgroup labels the cluster assignments are output as the estimated subgroup labels.