Inertial sensors within a head mounted display are used to track movement of the head mounted display. The tracked movement of the head mounted display is correlated to an action within a virtual reality scene that is currently displayed to a user wearing the head mounted display. The action within the virtual reality scene is based on a context of the virtual reality scene that is currently displayed. The detected movement of the head mounted display can be combined with other sensor data, such as gaze detection data, to determine the action within the virtual reality scene. In this manner, movements of the user as detected using the inertial sensors within the head mounted display are used as inputs to cause actions within the current context of the virtual reality scene as displayed within the head mounted display to the user.