A medical portable terminal device comprises: a console panel unit which has a touch panel for a first operator positioned in a non-sterile region to operate and allows control of operations of a plurality of medical instruments which are positioned in the non-sterile region; an image capture unit which captures an image of a region wherein a second operator positioned in a sterile region carries out a gesture manipulation; a recognition unit which recognizes the gesture manipulation from the captured image; a manipulation list storage unit which stores a manipulation list for causing a plurality of operations which are associated with the recognition results to be carried out upon the plurality of medical instruments; and a control unit which switches between a control of the plurality of medical instruments based on the manipulations of the console panel and a control of the plurality of medical instruments based on the gesture manipulation.