A method and apparatus of processing data to support Augmented Reality (AR) are provided. In the method, an image file including at least a scene description stream, an object descriptor stream, and a visual stream including a real-world medium is received, the scene description stream including an AR node that represents information about at least one piece of AR content used to augment the real-world medium, information about an AR locator and an object descriptor Identifier (ID) is acquired from the AR node, the AR locator describing when the at least one piece of AR content appears and how the real-world medium is augmented with the at least one piece of AR content and the object indicator ID identifying an Object Descriptor (OD) that describes the at least one piece of AR content, and at least one elementary stream descriptor.