The disclosed computer-implemented method for directing adaptive camera systems may include (i) monitoring, via a camera that is part of a camera system, a person within a physical environment, (ii) detecting, by the camera system, a gesture performed by the person, (iii) determining, by the camera system, that the gesture indicates an element of the physical environment, and (iv) performing, by the camera system, an action directed at the element of the physical environment in response to determining, by the camera system, that the gesture indicates the element of the physical environment. Various other methods, systems, and computer-readable media are also disclosed.