A method and apparatus of mitigating multipath coupling interference in a wireless repeater system 10 includes measuring an error signal E(z) in a signal path of the repeater which contains the desired communication signals and the multipath coupling interference and determining a cepstrum cee[n] of the error signal E(z), wherein the cepstrum cee[n] is reflective of the open-loop gain G(z) of the system. The method further includes extracting the impulse response of the open-loop gain G(z) from the cepstrum cee[n] of the error signal E(z), calculating a plurality of cancellation filter coefficients using the extracted impulse response, and adaptively applying the plurality of filter coefficients to an adjustable filter to generate the cancellation signal necessary to mitigate the multipath coupling interference.