In one embodiment of the present invention, an action agenda determining apparatus for determining an agenda of action to be taken with reference to surrounding situation is provided. An action agenda determining apparatus includes a matching model storage unit for storing an action agenda determining model that has learned in advance relation between time-sequence of prescribed feature information related to human motion extracted from surrounding images and action agenda to be taken, and a model reference unit for forming the time-sequence of prescribed feature information from the surrounding motion images and referring to the action agenda determining model stored in the matching model storage unit, for determining the action agenda to be taken. Sound may be included as part of the feature information.