A system and method for controlling rate adjustment of audio data to prevent underflow or overflow. In a dual audio/video system, a device can receive two input transport streams. To prevent underflow or overflow of audio data when audio data from a first transport stream is displayed in accordance with a sample rate derived from a second transport stream, a control for rate adjustment is used to match the source sample rate with the display rate. This rate adjustment module can be designed to add or drop audio samples based on a time base comparison or on a STC, PTC, and display rate comparison.