Techniques for protecting information in an audio file are provided. The techniques include obtaining an audio file, detecting information bearing one or more segments in a speech signal, wherein the information comprises information sought for protection, encrypting the information sought for protection by scrambling the one or more segments using a scrambling filter, and selectively decrypting an amount of the encrypted information, wherein the amount of the encrypted information to be decrypted depends on user access privilege, and wherein selectively decrypting the amount of the encrypted information protects said amount of the encrypted information. Techniques are also provided for protecting information in an audio file.