The present invention provides a paper processing apparatus and an image forming apparatus. The paper processing apparatus includes: a discharging unit that discharges the paper onto a staple tray; a paper returning unit that reverses the paper discharged by the discharging unit and performs alignment in a conveying direction by abutting the paper against a rear end reference fence; a supporting unit that supports the discharging unit in a state capable of coming in contact with or separating from the paper; a contact and separation unit by which the discharging unit comes in contact with or separates from the paper; and a pressurizing unit that is moved by the contact and separation unit, does not load the discharging unit in a standby state where the discharging unit is held apart from the paper, and applies contact pressure in a state where the discharging unit is in contact with the paper.