The invention relates to a pedestrian recognition system based on a random forest support vector machine. The pedestrian recognition system comprises a characteristic extraction module, a clustering module, a random forest creating module and a scoring model module. The invention also relates to a pedestrian recognition processing method based on the random forest support vector machine. A similarity ranking way is used for replacing the comparison of traditional similarity absolute values, a threshold value does not need to be delimited, and an obtained ranking result is convenient for users to judge; and since multiple characteristics are required for establishing a random forest model and samples can not be subjected to mutual classification perfection only from apparent characteristics, a K-means clustering algorithm is adopted to replace a phenomenon that a sample category is manually given, and potential relationships among samples can be mined. The method and the system exhibit robustness on pedestrian posture change and can eliminate interferences from other types of samples when the similarity is calculated, a ranking result of RankSVM (Support Vector Machine) is in the top, and recognition accuracy can be improved when the similarity is calculated. Compared with traditional algorithms including MCC, RankSVM and the like listed in the prior art, the pedestrian recognition system is high in recognition accuracy.