The present invention is to provide a method and system using compressed data directly for target tracking and target classification in videos. The present invention uses a video imager to generate compressive measurements, and a random subsampling operator to compress the video data. It uses a Gaussian Mixture Model (GMM) for target detection and manual location of the target and putting a bounding box around the targets in the first frame is not required. It further applies a saliency-based algorithm to re-center the captured target. This re-centering process can be repeated multiple times and each application of re-centering will improve over the previous one. A pixel completion algorithm is used to fill in the missing pixels for the captured target area. A Sparse Representation Classification (SRC) for target classification. Both the target templates in a dictionary and captured targets are transformed to the frequency domain using Fast Fourier Transform (FFT).