Videos of a scene are processed for view synthesis. The videos are acquired by corresponding cameras arranged so that a view of each camera overlaps with the view of at least one other camera. For each current block, disparity vectors are obtained from neighboring blocks. A depth block is based on a corresponding reference depth image and the disparity vectors. A prediction block is generated based on the depth block using backward warping of a motion field. Then, predictive coding for the current block using the prediction block. Backward mapping can also be performed in the spatial domain.