In one embodiment, a method includes receiving touch inputs from a user corresponding to an activation trigger for an assistant system executing on a head-mounted device at the head-mounted device, accessing signals from inertial measurement unit (IMU) sensors of the head-mounted device by the head-mounted device, determining that the user is either donning or doffing the head-mounted device by an on-device don/doff detection model and based only on the signals from the IMU sensors, and overriding the activation trigger to prevent an activation of the assistant system responsive to the received touch inputs.