A system and computer readable storage medium for creating an in-memory physical dictionary for data compression are provided. A new heuristic is defined for converting each of a plurality of logical nodes into a corresponding physical node forming a plurality of physical nodes. Each of the physical nodes are placed into the physical dictionary while traversing the dictionary tree in descending visit count order. Each physical node is placed in its nearest ascendant's cache-line with sufficient space. If there is no space in any of the ascendant's cache-line, then the physical node is placed into a new cache-line, unless a pre-defined packing threshold has been reached, in which case the physical node is placed in the first available cache-line.