A pump torque control system is capable of preventing hunting due to interference between speed sensing control and control of an engine speed of a prime mover when the temperature of a hydraulic fluid is low.A maximum absorption torque is set in a regulator 31 that controls displacement volumes of hydraulic pumps 2 and 3 based on a deviation between target and actual engine speeds of a prime mover 1. A second modification factor calculating section 45 and a control gain modifying section 49, which are included in a controller 23 that performs speed sensing control to ensure that the maximum absorption torque of the hydraulic pumps 2 and 3 is reduced, change a control gain of the speed sensing control based on a value detected by the hydraulic temperature sensor 34 to ensure that the control gain is reduced as the temperature of the hydraulic fluid is reduced.