Systems and methods for online augmentation for learned grasping are provided. In one embodiment, a method is provided that includes identifying an action from a discrete action space. The method includes identifying a second set of grasps of the agent utilizing a transition model based on the action and at least one contact parameter. The at least one contact parameter defines allowed states of contact for the agent. The method includes applying a reward function to evaluate each grasp of the second set of grasps based on a set of contact forces within a friction cone that minimizes a difference between an actual net wrench on the object and a predetermined net wrench. The reward function is optimized online using a lookahead tree. The method includes selecting a next grasp from the second set. The method includes causing the agent to execute the next grasp.