The present synthesizer generates an underlying spectrum, pitch and loudness for a sound to be synthesized, and then combines the underlying spectrum, pitch and loudness with stored Spectral, Pitch, and Loudness Fluctuations and noise elements. The input to the synthesizer is typically a MIDI stream. A MIDI preprocess block processes the MIDI input and generates the signals needed by the synthesizer to generate output sound phrases. The synthesizer comprises a harmonic synthesizer block (which generates an output representing the tonal audio portion of the output sound), an Underlying Spectrum, Pitch, and Loudness (which takes pitch and loudness and uses stored algorithms to generate the slowly varying portion of the output sound) and a Spectral, Pitch, and Loudness Fluctuation portion (which generates the quickly varying portion of the output sound by selecting and combining Spectral, Pitch, and Loudness Fluctuation segments stored in a database). A specialized analysis process is used to derive the formulas used by the Underlying Spectrum, Pitch, and Loudness and to generate and store the Spectral, Pitch, and Loudness Fluctuation segments stored in the database.