Disclosed is a sound processing apparatus including a factorization unit and an extraction unit. The factorization unit is configured to factorize frequency information obtained by performing time-frequency transformation on sound signals of a plurality of channels into a channel matrix expressing properties in a channel direction, a frequency matrix expressing properties in a frequency direction, and a time matrix expressing properties in a time direction. The extraction unit is configured to compare the channel matrix with a threshold and extract components specified by a result of the comparison from the channel matrix, the frequency matrix, and the time matrix to generate the frequency information on a sound from a desired sound source.