A parking assist system of a vehicle includes a camera that, when disposed at the vehicle, has a field of view exterior of the vehicle. An image processor is operable to process image data captured by the camera to detect parking space markers indicative of a parking space and to identify empty or available parking spaces. The image processor includes a parking space detection algorithm that detects parking space markers by (i) extracting low level features from captured image data, (ii) classifying pixels as being part of a parking space line or not part of a parking space line, (iii) performing spatial line fitting to find lines in the captured images and to apply parking space geometry constraints, and (iv) detecting and selecting rectangles in the captured images.