A classification model building apparatus and a classification model building method thereof are provided. The classification model building apparatus introduces a clustering algorithm to assist in training a deep learning model for classification and takes a sum of a clustering loss function, a center concentration loss function and a classification loss function multiplied by different weights, respectively, as a total loss function for training the deep learning model. Based on the total loss function, the classification model building apparatus adjusts parameters of the deep learning model through a backpropagation algorithm to build a classification model.