A device for implementing a rate-distortion optimization (RDO)-based small transform coefficients suppression algorithm includes a processing module, a control block, and a switch. The processing module may be configured to process a prediction residual associated with a transform unit (TU) of a video frame and to generate a number of coefficients and a processed output. The control block may be configured to receive the coefficients and to generate a control signal based on the plurality of coefficients. The switch may be configured to select, as an output of the device, one of the processed output or an output of a zero-out block based on the control signal. The zero-out block may be configured to assign a zero value to each of the plurality of coefficients of the TU to suppress small transform coefficients.