A medical robotic system has a joint coupled to medical device or a slave manipulator or robotic arm adapted to hold and/or move the medical device for performing a medical procedure, and a control system for controlling movement of the joint according to user manipulation of a master manipulator. The control system includes at least one joint controller having a sliding mode control for reducing stick-slip behavior on its controlled joint during fine motions of the joint. The sliding mode control computes a distance to a sliding surface, computes a reaching law gain, and processes the distance and reaching law gain to generate a sliding mode control action that is in absolute value less that a maximum desired feedback control action. The sliding mode control action is then further processed to generate a feedback torque command for the joint motor.