Methods and systems for determining semantic information associated with objects are provided. An example method includes receiving information associated with an object and information associated with a contextual situation of a robotic device from the robotic device. For example, the information associated with the contextual situation may include information associated with an environment in which the robotic device is configured to operate. The method may further include performing a search of a database for information associated with the object, and receiving results of the search. The method may also include a computing system determining semantic information associated with a use of the object based on the results of the search. The use of the object may be based on the information associated with the contextual situation of the robotic device. According to the method, the semantic information may be stored as supplemental information associated with the object.