A personal audio device, such as a wireless telephone, includes an adaptive noise canceling (ANC) circuit that adaptively generates an anti-noise signal from a reference microphone signal that measures the ambient audio and an error microphone signal that measures the output of an output transducer plus any ambient audio at that location and injects the anti-noise signal at the transducer output to cause cancellation of ambient audio sounds. A processing circuit uses the reference and error microphone to generate the anti-noise signal, which can be generated by an adaptive filter operating at a multiple of the ANC coefficient update rate. Downlink audio can be combined with the high data rate anti-noise signal by interpolation. High-pass filters in the control paths reduce DC offset in the ANC circuits, and ANC coefficient adaptation can be halted when downlink audio is not detected.