An assistant driving system with video recognition includes a camera array capturing environment images for the environment around the vehicle, an image recognition unit identifying environment objects and their image, location, color, speed and direction from the environment images, an environment monitor and control unit receiving the information generated by the image recognition unit and creating the rebuild environment information for the vicinity of the vehicle as well as judging the space relationship between the vehicle and the environment objects to generate the warning information, a video recognition display unit showing a single image for the rebuild environment information, a vehicle warning unit performing the processes corresponding to the warning information, a vehicle driving unit generating the driving control command based on the rebuild environment information, and a vehicle controlling unit receiving the driving control command to control the throttle, brake or steering wheel of the vehicle.