A tracking device includes: an image sensor that obtains image information of an image present within an image plane formed via an optical system; a tracking unit that tracks a target within the image plane based upon the image information; a focus detection unit that detects a focus adjustment state of the optical system over at least one focus detection area set in the image plane; a focus adjustment unit that executes focus adjustment for the optical system based upon the focus adjustment state detected in the focus detection area corresponding to a position of the target; a decision-making unit that makes a decision as to whether or not the target is being tracked based upon results of tracking operation executed by the tracking unit; and a control unit that bans the focus adjustment based upon results of the decision made by the decision-making unit.