A method and apparatus for video coding using a coding mode belonging to a mode group comprising an Intra Block Copy (IBC) mode and an Intra mode are disclosed. According to the present invention, for both IBC and Intra mode, a same default scaling matrix is used to derive the scaling matrix for a current block. In another embodiment, for the current block with block size of M×N or N×M, and M greater than N, a target scaling matrix is derived from an M×M scaling matrix by down-sampling the M×M scaling matrix to an M×N or N×M scaling matrix.