Techniques are described for coding residual data of a prediction residual block with transform skipping. A transform may be skipped for a residual block when the residual block is coded using either a lossless coding mode or a lossy coding mode in a transform skip mode. According to the techniques, based on a transform being skipped for a residual block, a rotation unit included in a video encoder or a video decoder determines whether to rotate the residual block prior to coding residual data of the residual block. In some examples, a rotation value may be explicitly signaled between the video encoder and the video decoder. In other examples, the video encoder and the video decoder may each independently determine whether to rotate the residual block based on a type of boundary at two or more edges of the residual block.