Devices, systems and methods for sub-block based prediction are described. In a representative aspect, a method for video processing includes determining a block size constrain, making a determination, based on the block size constrain, about whether or not a merge affine mode and a non-merge affine mode are allowed for a video block in a video frame, and generating a bitstream representation of the video block based on the making the determination.