A smart home device (e.g., a voice assistant device) includes an audio control system that determines a set of one or more audio devices to include nearby devices that are capable of providing audio streams that are audibly detected by a microphone of the smart home device. The audio control system initiates a voice-interaction mode for operating the smart home device to receive voice commands from a user and provide audio output in response to the voice commands. The audio control system transmits an audio control signal to nearby devices that configures each nearby device to implement one or more of: reducing a volume level associated with the audio streams generated by the nearby devices while the smart home device is operating in the voice-interaction mode; and transmitting, to the smart home device, audio stream data associated with a current audio stream generated for audible output by the nearby device.