The invention discloses a video-based detection and recognition system and a method of vehicles. The system comprises a target characteristic database module, a moving target detection module and a vehicle recognition module. A vehicle sample target characteristic database is built to store data by virtue of the target characteristic database module. After a moving target zone is detected by the moving target detection module, the vehicle recognition module builds a search window and recognizes vehicles. The system and the method resolve the technical problems of poor vehicle distinguishing capability and low distinguishing accuracy of the video-based vehicle detection under the condition of obvious background movement, vehicle adhesion, vehicle occlusion and the like in the prior art. The video-based detection and recognition system and the method of the vehicles have the advantages of excellent vehicle distinguishing capability and high accuracy.