A voice authentication system using a removable voice ID card comprises: at server side, a voiceprint database for storing the voiceprints of all authorized users; a voiceprint updating means for updating the voiceprints in said voiceprint database; and a voiceprint digest generator for generating a voiceprint digest according to a request from a client; at client side, a voice ID card for storing the voiceprint of an authorized user; a validation means for validating the voiceprint in the voice ID card on the basis of the voiceprint digest from the server; an audio device for performing voice interaction with a user; and a voice authentication means for determining whether the voiceprint from said voice ID card is of the same speaker as the voice from said audio device. The present invention can significantly avoid the abuse of a voice ID card when it is lost or stolen by using the voiceprint digest stored at server side to verify the voiceprint in the voice ID card.