公开(公告)号：US12143605B2

公开(公告)日：2024-11-12

申请号：US17543126

申请日：2021-12-06

Applicant: Google LLC

Inventor： Rohit Singh ,  Debargha Mukherjee ,  Elliott Karpilovsky ,  Lester Lu

IPC: H04N19/184 ,  H04N19/124 ,  H04N19/61

Abstract: Transform modes are derived for inter-predicted blocks using side information. A prediction residual is generated for a current video block using a reference frame. Side information associated with one or both of the current video block or the reference frame is identified. A trained transform is determined from amongst multiple trained transforms based on the side information, in which each of the trained transforms is determined using individual side information types and combinations of the individual side information types and the side information represents values of one of the individual side information types or one of the combinations of the individual side information types. The prediction residual is transformed according to the trained transform, and data associated with the transformed prediction residual and the side information are encoded to a bitstream.

公开(公告)号：US20240323361A1

公开(公告)日：2024-09-26

申请号：US18678031

申请日：2024-05-30

Applicant: GOOGLE LLC

Inventor： Lester Lu ,  Debargha Mukherjee ,  Elliott Karpilovsky

IPC: H04N19/107 ,  H04N19/122 ,  H04N19/176 ,  H04N19/50

CPC classification number: H04N19/107 ,  H04N19/122 ,  H04N19/176 ,  H04N19/50

Abstract: Decoding video data includes, for a block encoded using a prediction mode, determining a transform mode for the block using the prediction mode. The transform mode is a first mode when the prediction mode is an inter-prediction mode and is a second mode when the prediction mode is an intra-prediction mode. The first mode is an available first transform type that is a combination of transforms selected from first fixed transforms and first learned transforms that each comprise a respective transformation matrix generated iteratively using blocks predicted using the inter-prediction mode. The second mode is an available second transform type that is a combination of transforms selected from second fixed transforms, which is a proper subset of the first fixed transforms, and a second learned transform comprising a transformation matrix that is generated iteratively using blocks predicted using the intra-prediction mode. Decoding the block uses the prediction and transform modes.

公开(公告)号：US11197004B1

公开(公告)日：2021-12-07

申请号：US16919507

申请日：2020-07-02

Applicant: Google LLC

Inventor： Rohit Singh ,  Debargha Mukherjee ,  Elliott Karpilovsky ,  Lester Lu

IPC: H04N19/184 ,  H04N19/124 ,  H04N19/61

Abstract: Transform modes are derived for inter-predicted blocks using side information available within a bitstream. An inter-predicted encoded video block and side information are identified within a bitstream. Based on the side information, a trained transform is determined for inverse transforming transform coefficients of the inter-predicted encoded video block from amongst multiple trained transforms. The transform coefficients of the inter-predicted encoded video block are inverse transformed according to the trained transform to produce a prediction residual. A video block is reconstructed using the prediction residual and the reference frame. The video block is then output within an output video stream for storage or display. To determine the trained transforms, a learning model uses individual side information types and combinations of the individual side information types processed against a training data set.

公开(公告)号：US20240098280A1

公开(公告)日：2024-03-21

申请号：US18272862

申请日：2021-01-19

Applicant: Google LLC

Inventor： Urvang Joshi ,  Yue Chen ,  Sarah Parker ,  Elliott Karpilovsky ,  Debargha Mukherjee

IPC: H04N19/176 ,  H04N19/30

CPC classification number: H04N19/176 ,  H04N19/30

Abstract: Image coding using guided machine learning restoration may include obtaining reconstructed frame data by decoding, obtaining a restored frame by restoring the reconstructed frame, and outputting the restored frame. Obtaining the restored frame may include obtaining a reconstructed block, obtaining guide parameter values, obtaining a restored block, and including the restored block in the restored frame. Obtaining the restored block may include inputting the reconstructed block to an input layer of a trained guided convolutional neural network, wherein the neural network is constrained such that an output layer has a defined cardinality of channels, obtaining, from the output layer, neural network output channel predictions, obtaining a guided neural network prediction as a linear combination of the guide parameter values and the neural network output channel predictions, and generating the restored block using the guided neural network prediction.

公开(公告)号：US20210409705A1

公开(公告)日：2021-12-30

申请号：US16912767

申请日：2020-06-26

Applicant: GOOGLE LLC

Inventor： Sarah Parker ,  Debargha Mukherjee ,  Yue Chen ,  Elliott Karpilovsky ,  Urvang Joshi

IPC: H04N19/119 ,  H04N19/12 ,  H04N19/61 ,  H04N19/176 ,  G06N20/00

Abstract: Transform-level partitioning of a prediction residual block is performed to improve compression efficiency of video data. During encoding, a prediction residual block is generated responsive to prediction-level partitioning performed against a video block, a transform block partition type to use is determined based on the prediction residual block, a non-recursive transform-level partitioning is performed against the prediction residual block according to the transform block partition type, and transform blocks generated as a result of the transform-level partitioning are encoded to a bitstream. During decoding, a symbol representative of the transform block partition type used to encode transform blocks is derived from the bitstream, inverse transformed blocks are produced by inverse transforming encoded video data associated with the prediction residual block, and the prediction residual block is reproduced according to the transform block partition type and used to reconstruct the video block, which is output within an output video stream.

公开(公告)号：US20210185312A1

公开(公告)日：2021-06-17

申请号：US16712057

申请日：2019-12-12

Applicant: GOOGLE LLC

Inventor： Lester Lu ,  Debargha Mukherjee ,  Elliott Karpilovsky

IPC: H04N19/107 ,  H04N19/122 ,  H04N19/50 ,  H04N19/176

Abstract: Coding a block of video data includes determining a prediction mode for the block, which is an inter-prediction or intra-prediction mode, determining a transform type for the block, and coding the block using the prediction mode and the transform type. The transform type is one of a first plurality of transform types when the prediction mode is the inter-prediction mode, and is one of a second plurality of transform types when the prediction mode is the intra-prediction mode. The first plurality of transform types includes first fixed transform types and first mode-dependent transform types that are based on a first learned transform generated using inter-predicted blocks. The second plurality of transform types includes second fixed transform types and second mode-dependent transform types that are based on a second learned transform generated using intra-predicted blocks. The first and second fixed transform types have at least some fixed transform types in common.

公开(公告)号：US12294705B2

公开(公告)日：2025-05-06

申请号：US17779692

申请日：2019-12-23

Applicant: Google LLC

Inventor： Debargha Mukherjee ,  Lester Lu ,  Elliott Karpilovsky

IPC: H04N19/126 ,  H04N19/119 ,  H04N19/12 ,  H04N19/176 ,  H04N19/42 ,  H04N19/463 ,  H04N19/60

Abstract: Residual coding using vector quantization (VQ) is described. A flag indicating whether a residual block for the current block is encoded using VQ. In response to the flag indicating that the residual block is encoded using VQ, a parameter indicating an entry in a codebook is decoded, and the residual block is decoded using the entry. In response to the flag indicating that the residual block is not encoded using VQ, the residual block is decoded based on a skip flag indicating whether the current block is encoded using transform skip. The current block is reconstructed using the residual block.

公开(公告)号：US20220094950A1

公开(公告)日：2022-03-24

申请号：US17543126

申请日：2021-12-06

Applicant: Google LLC

Inventor： Rohit Singh ,  Debargha Mukherjee ,  Elliott Karpilovsky ,  Lester Lu

IPC: H04N19/184 ,  H04N19/124 ,  H04N19/61

Abstract: Transform modes are derived for inter-predicted blocks using side information. A prediction residual is generated for a current video block using a reference frame. Side information associated with one or both of the current video block or the reference frame is identified. A trained transform is determined from amongst multiple trained transforms based on the side information, in which each of the trained transforms is determined using individual side information types and combinations of the individual side information types and the side information represents values of one of the individual side information types or one of the combinations of the individual side information types. The prediction residual is transformed according to the trained transform, and data associated with the transformed prediction residual and the side information are encoded to a bitstream.

公开(公告)号：US11284071B2

公开(公告)日：2022-03-22

申请号：US16712057

申请日：2019-12-12

Applicant: GOOGLE LLC

Inventor： Lester Lu ,  Debargha Mukherjee ,  Elliott Karpilovsky

IPC: H04N19/107 ,  H04N19/122 ,  H04N19/176 ,  H04N19/50

Abstract: Coding a block of video data includes determining a prediction mode for the block, which is an inter-prediction or intra-prediction mode, determining a transform type for the block, and coding the block using the prediction mode and the transform type. The transform type is one of a first plurality of transform types when the prediction mode is the inter-prediction mode, and is one of a second plurality of transform types when the prediction mode is the intra-prediction mode. The first plurality of transform types includes first fixed transform types and first mode-dependent transform types that are based on a first learned transform generated using inter-predicted blocks. The second plurality of transform types includes second fixed transform types and second mode-dependent transform types that are based on a second learned transform generated using intra-predicted blocks. The first and second fixed transform types have at least some fixed transform types in common.

公开(公告)号：US12003706B2

公开(公告)日：2024-06-04

申请号：US17700298

申请日：2022-03-21

Applicant: Google LLC

Inventor： Lester Lu ,  Debargha Mukherjee ,  Elliott Karpilovsky

IPC: H04N19/107 ,  H04N19/122 ,  H04N19/176 ,  H04N19/50

CPC classification number: H04N19/107 ,  H04N19/122 ,  H04N19/176 ,  H04N19/50

Abstract: Decoding video data includes, for a block encoded using a prediction mode, determining a transform mode for the block using the prediction mode. The transform mode is a first mode when the prediction mode is an inter-prediction mode and is a second mode when the prediction mode is an intra-prediction mode. The first mode is an available first transform type that is a combination of transforms selected from first fixed transforms and first learned transforms that each comprise a respective transformation matrix generated iteratively using blocks predicted using the inter-prediction mode. The second mode is an available second transform type that is a combination of transforms selected from second fixed transforms, which is a proper subset of the first fixed transforms, and a second learned transform comprising a transformation matrix that is generated iteratively using blocks predicted using the intra-prediction mode. Decoding the block uses the prediction and transform modes.