Methods, apparatus, and computer readable media are described related to utilizing a context of an ongoing human-to-computer dialog to enhance the ability of an automated assistant to interpret and respond when a user abruptly transitions between different domains (subjects). In various implementations, natural language input may be received from a user during an ongoing human-to-computer dialog with an automated assistant. Grammar(s) may be selected to parse the natural language input. The selecting may be based on topic(s) stored as part of a contextual data structure associated with the ongoing human-to-computer dialog. The natural language input may be parsed based on the selected grammar(s) to generate parse(s). Based on the parse(s), a natural language response may be generated and output to the user using an output device. Any topic(s) raised by the parse(s) or the natural language response may be identified and added to the contextual data structure.