A package structure and a method of forming the same are provided. The package structure includes a first die, a second die, a first encapsulant, and a buffer layer. The first die and the second die are disposed side by side. The first encapsulant encapsulates the first die and the second die. The second die includes a die stack encapsulated by a second encapsulant encapsulating a die stack. The buffer layer is disposed between the first encapsulant and the second encapsulant and covers at least a sidewall of the second die and disposed between the first encapsulant and the second encapsulant. The buffer layer has a Young's modulus less than a Young's modulus of the first encapsulant and a Young's modulus of the second encapsulant.