Systems and techniques for enhancing the physical games with augmented reality is described. A system identifies a characteristic of a physical object that is located on a physical surface. The system obtains, based on the characteristic, information indicative of a rule set for the game and a state of the game based on the characteristic. A game engine is initialized based on the rule set and game state. The game engine generates a subsequent state for the game. The system renders, based on the subsequent state, an augmented reality display that incorporates the physical surface and physical object.