An apparatus and method for providing primitive visual knowledge are disclosed. The method of providing primitive visual knowledge includes receiving an image in a form of a digital image sequence, dividing the received image into scenes, extracting a representative shot from each of the scenes, extracting objects from frames which compose the representative shot, extracting action verbs based on a mutual relationship between the extracted objects, selecting a frame best expressing the mutual relationship with the objects, which are the basis for the extracting of the action verbs, as a key frame, generating the primitive visual knowledge based on the selected key frame, storing the generated primitive visual knowledge in a database, and visualizing the primitive visual knowledge stored in the database to provide the primitive visual knowledge to a manager.