Techniques are discussed herein for executing log-based driving simulations to evaluate the performance and functionalities of vehicle control systems. A simulation system may execute a log-based driving simulation including playback agents whose behavior is based on the log data captured by a vehicle operating in an environment. The simulation system may determine interactions associated with the playback agents, and may convert the playback agents to smart agents during the driving simulation. During a driving simulation, playback agents that have been converted to smart agents may interact with additional playback agents, causing a cascading effect of additional conversions. Converting playback agents to smart agents may include initiating a planning component to control the smart agent, which may be based on determinations of a destination and/or driving attributes based on the playback agent.