Techniques are disclosed for coordinating interactions between a user device and a plurality of accessory devices. In one example, a user device receives information identifying one or more accessory devices in communication with the user device. The user device may implement accessory interaction instances for each of the identified accessories. A first accessory interaction instance can be associated with a first accessory among the identified accessories and receive a first audio input from the first accessory corresponding to a user request. The first accessory interaction instance can process a portion of the received audio input and receive a first response from a server computer. The user device may then transmit the first response to the first accessory device.