An abnormality detection system is applied to an autonomous driving system of a target vehicle. The abnormality detection system stores travel plan information and reference travel information. The travel plan information and the reference travel information indicate a travel plan of the target vehicle and a travel record of a reference vehicle different from the target vehicle, respectively. The abnormality detection system calculates a deviation between the travel plan of the target vehicle and the travel record of the reference vehicle for each determination position based on the travel plan information and the reference travel information. The abnormality detection system extracts the determination position at which the deviation exceeds a threshold as an abnormal position related to an abnormality of the autonomous driving system. The abnormality detection system enables to appropriately detect the abnormality of the autonomous driving system independently of the recognition performance of the autonomous driving system.