In processing a game scene for display, in one embodiment input controller position information from a host memory is provided directly to a graphics processor rather than first being processed by a 3D application in a host processor. This results in more direct and timely processing of position information and reduces the number of 3D processing pipeline steps the controller position information must pass through thus reducing the user's perceived latency between moving the input controller and seeing the displayed results. In another embodiment, the input controller position information is provided directly from an input controller to a graphics card or subsystem rather than first going through a host processor or memory. This results in even more direct and timely processing of position information by further reducing the number of 3D processing pipeline steps the controller position information must pass through thus further reducing the user's perceived latency.