A semiconductor device is made by providing a sacrificial substrate and depositing an adhesive layer over the sacrificial substrate. A first conductive layer is formed over the adhesive layer. A polymer pillar is formed over the first conductive layer. A second conductive layer is formed over the polymer pillar to create a conductive pillar with inner polymer core. A semiconductor die or component is mounted over the substrate. An encapsulant is deposited over the semiconductor die or component and around the conductive pillar. A first interconnect structure is formed over a first side of the encapsulant. The first interconnect structure is electrically connected to the conductive pillar. The sacrificial substrate and adhesive layers are removed. A second interconnect structure is formed over a second side of the encapsulant opposite the first interconnect structure. The second interconnect structure is electrically connected to the conductive pillar.