Systems and methods for estimating velocity of an autonomous vehicle and state information of a surrounding vehicle are provided. In some aspects, the system includes a memory that stores instructions for executing processes for estimating velocity of an autonomous vehicle and state information of the surrounding vehicle and a processor configured to execute the instructions. In various aspects, the processes include: receiving image data from an image capturing device; performing a ground plane estimation by predicting a depth of points on a road surface based on an estimated pixel-level depth; determining a three-dimensional (3D) bounding box of the surrounding vehicle; determining the state information of the surrounding vehicle based on the ground plane estimation and the 3D bounding box; and determining the velocity of the autonomous vehicle based on an immovable object relative to the autonomous vehicle. In some aspects, an operation of the autonomous vehicle may be controlled based on at least one of the state information or the velocity of the autonomous vehicles.