A process to upgrade heavy oil and convert the heavy oil into lower boiling hydrocarbon products is provided. The process employs a catalyst slurry comprising catalyst particles with an average particle size ranging from 1 to 20 microns. In the upgrade process, spent slurry catalyst in heavy oil is generated as an effluent stream. In one aspect, the process further includes the recovery of spent catalyst by separating heavy oil from catalyst particles in the slurry. In one embodiment, slurry catalyst in heavy oil is combined with solvent to form a combined slurry-solvent stream. The combined slurry-solvent stream is filtered in a deoiling zone using membrane filtration. Hydrocarbons, i.e., solvent and residual heavy oil, can be subsequently separated from catalyst particles in a drying zone. Valuable metals can be recovered from catalyst particles for subsequent re-use in a catalyst synthesis unit, generating fresh slurry catalyst.