An image-based vehicle classification system includes a camera and an image server connected to the camera. The camera captures images of a road to result in an image stream. The image server includes a processor for receiving the image stream from the camera. For each of the images of the image stream, the processor performs image segmentation, a thinning process, an erosion process and a dilation process, and classifies, by a neural network classifier, a vehicle image part contained in the image into one of a large-size car class, a passenger car class and a motorcycle class when it is determined that the vehicle image is crossing an imaginary line set in advance in the image for counting vehicles.