A tool bit holder assembly includes a shank receivable in a tool holder of a rotary tool, and a body defining a socket that removably receives a tool bit. An annular recess, defined in an exterior surface of the body, has a front and rear shoulders and an axial length. A floating sleeve with a ring magnet at its front end is slidably received over the body with an elastic retainer movably received in the annular recess. The elastic retainer has an axial length less than the axial length of the annular recess to enable the sleeve to float axially between a rear position where the retainer abuts the rear shoulder and a front position where the retainer abuts the front shoulder. The sleeve is axially moveable rearward from the rear position to a parked position where the elastic retainer engages the body rearward of the rear shoulder.