Temporal foveated rendering using motion estimation

    公开(公告)号:US10169843B1

    公开(公告)日:2019-01-01

    申请号:US15818072

    申请日:2017-11-20

    摘要: A processing system selectively renders pixels or blocks of pixels of an image and leaves some pixels or blocks of pixels unrendered to conserve resources. The processing system generates a motion vector field to identify regions of an image having moving areas. The processing system uses a rendering processor to identify as regions of interest those units having little to no motion, based on the motion vector field, and a large amount of edge activity, and to minimize the probability of unrendered pixels, or “holes”, in these regions. To avoid noticeable patterns, the rendering processor applies a probability map to determine the possible locations of holes, assigning to each unit a probability indicating the percentage of pixels within the unit that will be holes, and assigning a lower probability to units identified as regions of interest.

    Perceptual importance maps for image processing

    公开(公告)号:US11475653B2

    公开(公告)日:2022-10-18

    申请号:US15965640

    申请日:2018-04-27

    发明人: Boris Ivanovic

    摘要: The present disclosure is directed to techniques for determining a perceptual importance map. The perceptual importance map indicates the relative importance to the human visual system of different portions of an image. The techniques include obtaining cost values for the blocks of an image, where cost values are values used in determining motion vectors. For each block, a confidence value is derived from the cost values. The confidence value indicates the confidence with which the motion vector is believed to be correct. A perceptual importance value is determined based on the confidence value via one or more modifications to the confidence value to better reflect importance to the human visual system. The generated perceptual importance values can be used for various purposes such as allocating bits for encoding, identifying regions of interest, or selectively rendering portions of an image with greater or lesser detail based on relative perceptual importance.

    Determining quality values for blocks of encoded video

    公开(公告)号:US11310496B2

    公开(公告)日:2022-04-19

    申请号:US16366959

    申请日:2019-03-27

    摘要: A technique for determining a quality value for a subject block of encoded video is provided. Contributing blocks, of the same frame and/or different frames of the subject block, are determined by identifying blocks likely to be a part of the same moving object or background as the subject block. A spatial and/or temporal filter is then applied to the quality values of the contributing blocks and an initial quality value of the subject block. With a spatial filter, quality values for contributing blocks from the same frame are combined and used to modify the quality value of the subject block. With a spatial filter, a temporal characteristic quality value for contributing blocks of one or more other frames (such as the immediately previous frame) is determined and then combined with a quality value representative of the subject block.

    Block type prediction leveraging block-based pixel activities

    公开(公告)号:US11234004B2

    公开(公告)日:2022-01-25

    申请号:US16207893

    申请日:2018-12-03

    摘要: Systems, apparatuses, and methods for block type prediction leveraging block-based pixel activities are disclosed. A pre-encoder generates predictions of block types for the blocks of a video frame based on associated pixel activities. For each block, the pre-encoder calculates the difference between the pixel activities of the block of a current frame and the pixel activities of a corresponding block of a previous video frame. If the difference is less than a first threshold, the pre-encoder predicts that the block will be a skip block. If the difference is in between the first threshold and a second threshold, the pre-encoder predicts that the block will be a P-block. Otherwise, if the difference is greater than the second threshold, then the pre-encoder predicts that the block will be an I-block. The pre-encoder uses the predictions to select quantization parameter (QP) ranges for encoding the blocks of the video frame.

    Multi-scale metric-based encoding

    公开(公告)号:US11102488B2

    公开(公告)日:2021-08-24

    申请号:US16427482

    申请日:2019-05-31

    摘要: A processing system analyzes pixel activity levels of blocks of a picture at a plurality of spatial scales and/or dynamic ranges to generate a multi-scale metric that indicates how bit allocation or assignment of a given quantization parameter (QP) will affect the perceptual quality of the block. Blocks that have similar multi-scale metrics are likely to be visually similar and to benefit from similar bit allocations or QPs. Based on the multi-scale metric, an encoder encodes each block of the picture with a QP and/or a number of bits.

    PERCEPTUAL IMPORTANCE MAPS FOR IMAGE PROCESSING

    公开(公告)号:US20190332883A1

    公开(公告)日:2019-10-31

    申请号:US15965640

    申请日:2018-04-27

    发明人: Boris Ivanovic

    摘要: The present disclosure is directed to techniques for determining a perceptual importance map. The perceptual importance map indicates the relative importance to the human visual system of different portions of an image. The techniques include obtaining cost values for the blocks of an image, where cost values are values used in determining motion vectors. For each block, a confidence value is derived from the cost values. The confidence value indicates the confidence with which the motion vector is believed to be correct. A perceptual importance value is determined based on the confidence value via one or more modifications to the confidence value to better reflect importance to the human visual system. The generated perceptual importance values can be used for various purposes such as allocating bits for encoding, identifying regions of interest, or selectively rendering portions of an image with greater or lesser detail based on relative perceptual importance.

    Video frame rate conversion using streamed metadata

    公开(公告)号:US10412462B2

    公开(公告)日:2019-09-10

    申请号:US15346392

    申请日:2016-11-08

    发明人: Boris Ivanovic

    摘要: A video server generates metadata representative of interpolation parameters for portions of a first frame representative of a scene in a stream of frames including the first frame. The interpolation parameters are used to generate at least one interpolated frame representative of the scene subsequent to the first frame and prior to a second frame in the stream of frames. The video server incorporates the metadata into the stream and transmits the stream including the multiplexed metadata. A video client receives the first frame representative the stream of frames including the metadata. The video client generates one or more interpolated frames representative of the scene subsequent to the first frame and prior to a second frame in the stream of frames based on the first frame and the metadata. The video client displays the first frame, the one or more interpolated frames, and the second frame.

    Motion estimation using pixel activity metrics

    公开(公告)号:US12132923B2

    公开(公告)日:2024-10-29

    申请号:US16113212

    申请日:2018-08-27

    IPC分类号: H04N11/02 H04N19/513

    CPC分类号: H04N19/513

    摘要: A processing system estimates motion between blocks of images based on differences in pixel activities between blocks. Blocks having similar pixel activities are more likely to be matches for motion estimation than are blocks having dissimilar pixel activities. Accordingly, the processing system compares pixel activity between current block of a current image and candidate blocks within a search area in the reference image, and estimates motion based on a subset of candidate blocks having a difference in pixel activity from that of the current block within a threshold.

    Benefit-based bitrate distribution for video encoding

    公开(公告)号:US11997275B2

    公开(公告)日:2024-05-28

    申请号:US16113302

    申请日:2018-08-27

    CPC分类号: H04N19/124 H04N19/154

    摘要: A multimedia system allocates, during encoding of a multimedia stream, bits to portions of frames based on quality metrics and bit usages for different quantization parameters (QPs). An encoder of the multimedia system encodes a frame in a first pass with a first QP and in a second pass with a second QP. A comparator of the multimedia system measures and compares quality metrics, such as mean squared error, for each portion of the frame encoded with the first QP and the second QP. The comparator compares the difference between the quality metrics for each portion encoded with each QP to a threshold. If the difference in quality metrics for a portion exceeds the threshold, the comparator selects the portion for inclusion in a subset of portions to be encoded with the second QP.

    Video encode pre-analysis bit budgeting based on context and features

    公开(公告)号:US11843772B2

    公开(公告)日:2023-12-12

    申请号:US16706473

    申请日:2019-12-06

    摘要: Systems, apparatuses, and methods for bit budgeting in video encode pre-analysis based on context and features are disclosed. A pre-encoder receives a video frame and evaluates each block of the frame for the presence of several contextual indicators. The contextual indicators can include memory colors, text, depth of field, and other specific objects. For each contextual indicator detected, a coefficient is generated and added with other coefficients to generate a final importance value for the block. The coefficients can be adjusted so that only a defined fraction of the picture is deemed important. The final importance value of the block is used to determine the bit budget for the block. The block bit budgets are provided to the encoder and used to influence the quantization parameters used for encoding the blocks.