A method and apparatus for video encoding and decoding with prediction offset in a video coding system are disclosed. A prediction offset for a current block is derived from a reconstructed residual signal of one or more coded blocks. At the encoder side, a residual signal for the current block is calculated by subtracting a prediction signal and the prediction offset from an original signal of the current block, and the residual signal is encoded to generate an encoded bitstream. At the decoder side, a residual signal for the current block is obtained from input data, a reconstructed signal for the current block is calculated by adding a prediction signal and the prediction offset to the residual signal, and the current block is decoded using the reconstructed signal.