Devices, systems and methods for digital video coding, which includes matrix-based intra prediction methods for video coding, are described. In a representative aspect, a method for video processing includes performing a first determination whether a luma video block of a video is coded using a matrix based intra prediction (MIP) mode; performing a second determination that the luma video block is applicable for determining a chroma intra mode for a current chroma video block of the video; performing, based on the first determination and the second determination, a third determination about the chroma intra mode to be used for the current chroma video block; and performing, based on the third determination, a conversion between the current chroma video block and a bitstream representation of the current chroma video block.