The invention discloses a visual tracking algorithm taking paper as an object, and the algorithm comprises the steps: carrying out the classification of positive and negative samples in a first image frame; calculating the feature values of all positive and negative samples, and determining a first target area; carrying out the sampling nearby the target region of the former frame on each image frame after the second frame through a sampling block; calculating the scores of all sampling blocks through combining with the feature values of the positive and negative samples of the former frame; employing the sampling block with the highest score, and carrying out the detection of each sampling block through an LSD linear detection algorithm; finding the sampling blocks forming an inclined angle in the section [70 degrees, 110 degrees], wherein the intersection of two straight lines is located in the sampling block; further selecting one sampling blocks with the intersection of two straight lines being the nearest to the center as the target sampling block. The method gives consideration to a problem of a photographing angle of a camera when paper moves, and enables a finally obtained tracking target to be more accurate.