In a personal audio system (100), a remote controller (120) has a touch-sensitive area (122) and is worn in or by a human ear. A temporal pattern in the area (122), which is being touched, is detected and used for remotely controlling a device (110) for personal audio by means of a control signal (130). This prevents the hassle involved in finding, manipulating and operating a conventional remote control that is typically dangling somewhere along a wire.