The invention discloses a camera based virtual vehicle driving system and a virtual driving method. The driving system comprises calibration paper, a camera, a controller, a parameter set unit and a display, wherein the calibration paper has a plurality of vehicle control information patterns to be touched by virtual drivers and can be controlled by rotating; the camera can shoot state information of the vehicle control information patterns on the calibration paper in real time; the controller identifies and analyzes the state information shot by the camera; and the parameter set unit is connected with the controller. The method comprises the steps of driving preparation, calibration paper control, visual information acquisition, visual information interaction and driving control, which is to carry out control of speed changing, honking, turning, braking and backing on a vehicle by controlling the calibration paper. The system has novel and reasonable design, low investment cost, simple and convenient use and operation, good virtual driving effect and strong operability, and realizes virtual driving on the virtual vehicle by using the camera as an interaction mode and controlling the calibration paper in an actual environment.