A tool for use with a surgical robotic manipulator that comprises an energy applicator including a shaft extending along an axis between a proximal end and a distal end. The shaft has an axial-force receiving surface. A tool assembly comprises a support structure to support the energy applicator, an axial connector assembly arranged to engage and releasably lock the energy applicator to the support structure in a locked state, a drive system coupled to the support structure to rotatably drive the shaft of the energy applicator about the axis, a collet assembly cooperating with the axial connector assembly and configured to apply a force to the axial-force receiving surface of the energy applicator in the locked state, and a reference surface. The force includes an axial component directing the energy applicator proximally into continuous contact with the reference surface in the locked state.