PROF (Prediction Refinement with Optical Flow) and BDOF (Bi-directional Optical Flow) use the optical flow model to provide sample-wise motion compensation refinement. In one implementation, to calculate the spatial gradients in BDOF, an extended region is generated around a block to be coded. To simplify the computation, BDOF may use the same padding operations as in PROF, namely, a padding sample is copied from the nearest integer neighbor in the reference picture. BDOF and PROF can also be used together when sub-block based affine motion-compensated prediction is used. BDOF can be applied after PROF generates new prediction blocks, or BDOF can use the spatial gradients in the original prediction blocks to re-use the computations with PROF. In addition, BDOF and PROF operations can be combined, and the prediction adjustment will be directly based on the spatial gradients in the original prediction blocks.