公开(公告)号：US12190403B2

公开(公告)日：2025-01-07

申请号：US17792062

申请日：2020-01-13

Applicant: GOOGLE LLC

Inventor： Ruohan Zhan ,  Feng Yang ,  Xiyang Luo ,  Peyman Milanfar ,  Huiwen Chang ,  Ce Liu

IPC: G06T1/00 ,  G06T9/00 ,  G06V10/75 ,  G06V10/82

Abstract: Methods, systems, and computer programs encoded on a computer storage medium, that relate to extracting digital watermarks from images, irrespective of distortions introduced into these images. Methods can include inputting a first data item into a channel encoder that can generate a first encoded data item that is greater in length than the first data item and that (1) includes the input data item and (2) new data this is redundant of the input data item. Based on the first encoded data item and a first image, an encoder model can generate a first encoded image into which the first encoded data is embedded as a digital watermark. A decoder model can decode the first encoded data item to generate a second data, which can be decoded by the channel decoder to generate data that is predicted to be the first data.

公开(公告)号：US20240022760A1

公开(公告)日：2024-01-18

申请号：US18256837

申请日：2021-08-05

Applicant: Google LLC

Inventor： Yinxiao Li ,  Peyman Milanfar ,  Feng Yang ,  Ce Liu ,  Ming-Hsuan Yang ,  Pengchong Jin

IPC: H04N19/59 ,  G06T3/00 ,  H04N19/117 ,  G06V10/74 ,  H04N19/503 ,  H04N19/70 ,  H04N19/80

CPC classification number: H04N19/59 ,  G06T3/0093 ,  H04N19/117 ,  G06V10/761 ,  H04N19/503 ,  H04N19/70 ,  H04N19/80

Abstract: Example aspects of the present disclosure are directed to systems and methods which feature a machine-learned video super-resolution (VSR) model which has been trained using a bi-directional training approach. In particular, the present disclosure provides a compression-informed (e.g., compression-aware) super-resolution model that can perform well on real-world videos with different levels of compression. Specifically, example models described herein can include three modules to robustly restore the missing information caused by video compression. First, a bi-directional recurrent module can be used to reduce the accumulated warping error from the random locations of the intra-frame from compressed video frames. Second, a detail-aware flow estimation module can be added to enable recovery of high resolution (HR) flow from compressed low resolution (LR) frames. Finally, a Laplacian enhancement module can add high-frequency information to the warped HR frames washed out by video encoding.

公开(公告)号：US20180253826A1

公开(公告)日：2018-09-06

申请号：US15970393

申请日：2018-05-03

Applicant: Google LLC

Inventor： Peyman Milanfar ,  Yaniv Romano

IPC: G06T3/40 ,  G06T5/50 ,  G06T5/00 ,  G06K9/46

CPC classification number: G06T3/4053 ,  G06K9/4604 ,  G06T3/40 ,  G06T3/403 ,  G06T5/003 ,  G06T5/50 ,  G06T2207/20024 ,  G06T2207/20081 ,  G06T2207/20221 ,  G06T2210/36

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for upscaling an image. One of the methods includes upscaling a low resolution image, creating first pixel subsets of the first upscaled image, creating second pixel subsets of a high resolution image, determining, for each subset in the pixel subsets, a value of a property of the pixel subset, determining, for each subset in the pixel subsets, a group of subsets to which the corresponding pixel subset belongs using the value of the property, and determining, for each of the groups of subsets, a filter to apply to each of the first pixel subsets that correspond to the pixel subsets in the group to create a final pixel subset that approximates the corresponding second pixel subset using the first pixel subset, a combination of all of the final pixel subsets representing a second upscaled image.

公开(公告)号：US20250069382A1

公开(公告)日：2025-02-27

申请号：US18726881

申请日：2023-01-05

Applicant: Google LLC

Inventor： Yinxiao Li ,  Zhengzhong Tu ,  Hossein Talebi ,  Han Zhang ,  Feng Yang ,  Peyman Milanfar

IPC: G06V10/82 ,  G06V10/764 ,  G06V10/77

Abstract: Provided are machine learning systems and models featuring resolution-flexible multi-axis attention blocks. In particular, the present disclosure provides example multi-axis MLP based architectures (example implementations of which can be generally referred to as MAXIM) that can serve as an efficient and flexible general-purpose vision backbone for image processing tasks. In some implementations, MAXIM can use a UNet-shaped hierarchical structure and supports long-range interactions enabled by spatially-gated MLPs. Specifically, some example implementations of MAXIM can contain two MLP-based building blocks: a multi-axis gated MLP that allows for efficient and scalable spatial mixing of local and global visual cues, and a cross-gating block, an alternative to cross-attention, which accounts for cross-feature mutual conditioning.

公开(公告)号：US12206914B2

公开(公告)日：2025-01-21

申请号：US18021636

申请日：2022-06-08

Applicant: Google LLC

Inventor： Yilin Wang ,  Balineedu Adsumilli ,  Junjie Ke ,  Hossein Talebi ,  Joong Yim ,  Neil Birkbeck ,  Peyman Milanfar ,  Feng Yang

IPC: H04N21/266 ,  G06N3/045 ,  H04N17/02 ,  H04N19/154 ,  H04N21/234 ,  H04N21/434 ,  H04N21/44 ,  H04N21/466

Abstract: Methods, systems, and media for determining perceptual quality indicators of video content items are provided. In some embodiments, the method comprises: receiving a video content item; extracting a plurality of frames from the video content item; determining, using a first subnetwork of a deep neural network, a content quality indicator for each frame of the plurality of frames of the video content item; determining, using a second subnetwork of the deep neural network, a video distortion indicator for each frame of the plurality of frames of the video content item; determining, using a third subnetwork of the deep neural network, a compression sensitivity indicator for each frame of the plurality of frames of the video content item; generating a quality level for each frame of the plurality of frames of the video content item that concatenates the content quality indicator, the video distortion indicator, and the compression sensitivity indicator for that frame of the video content item; generating an overall quality level for video content item by aggregating the quality level of each frame of the plurality of frames; and causing a video recommendation to be presented based on the overall quality level of the video content item.

公开(公告)号：US20230267307A1

公开(公告)日：2023-08-24

申请号：US18014314

申请日：2020-07-23

Applicant: Google LLC

Inventor： Qifei Wang ,  Junjie Ke ,  Grace Chu ,  Gabriel Mintzer Bender ,  Luciano Sbaiz ,  Feng Yang ,  Andrew Gerald Howard ,  Alec Michael Go ,  Jeffrey M. Gilbert ,  Peyman Milanfar ,  Joshua William Charles Greaves

IPC: G06N3/045 ,  G06N3/092 ,  G06N3/084

CPC classification number: G06N3/045 ,  G06N3/084 ,  G06N3/092

Abstract: Systems and methods of the present disclosure are directed to a method for generating a machine-learned multitask model configured to perform tasks. The method can include obtaining a machine-learned multitask search model comprising candidate nodes. The method can include obtaining tasks and machine-learned task controller models associated with the tasks. As an example, for a task, the method can include using the task controller model to route a subset of the candidate nodes in a machine-learned task submodel for the corresponding task. The method can include inputting task input data to the task submodel to obtain a task output. The method can include generating, using the task output, a feedback value based on an objective function. The method can include adjusting parameters of the task controller model based on the feedback value.

公开(公告)号：US20230224596A1

公开(公告)日：2023-07-13

申请号：US18184179

申请日：2023-03-15

Applicant: Google LLC

Inventor： Yi Hung Chen ,  Chia-Kai Liang ,  Bartlomiej Maciej Wronski ,  Peyman Milanfar ,  Ignacio Garcia Dorado

IPC: G06T3/40 ,  H04N23/68

CPC classification number: G06T3/4069 ,  H04N23/6811 ,  H04N23/687

Abstract: The present disclosure describes systems and techniques directed to optical image stabilization movement to create a super-resolution image of a scene. The systems and techniques include a user device (102) introducing (502), through an optical image stabilization system (114), movement to one or more components of a camera system (112) of the user device (102). The user device (102) then captures (504) respective and multiple frames (306) of an image of a scene, where the respective and multiple frames (306) of the image of the scene have respective, sub-pixel offsets of the image of the scene across the multiple frames (306) as a result of the introduced movement to the one or more components of the camera system (112). The user device (102) performs (506), based on the respective, sub-pixel offsets of the image of the scene across the respective, multiple frames (306), super-resolution computations and creates (508) the super-resolution image of the scene based on the super-resolution computations.

公开(公告)号：US20230222623A1

公开(公告)日：2023-07-13

申请号：US17787699

申请日：2021-07-01

Applicant: Google LLC

Inventor： Junjie Ke ,  Feng Yang ,  Qifei Wang ,  Yilin Wang ,  Peyman Milanfar

IPC: G06T3/00 ,  G06T3/40 ,  G06T7/00

CPC classification number: G06T3/0012 ,  G06T3/40 ,  G06T7/0002 ,  G06T2207/30168 ,  G06T2207/20081 ,  G06T2207/20016

Abstract: The technology employs a patch-based multi-scale Transformer (300) that is usable with various imaging applications. This avoids constraints on image fixed input size and predicts the quality effectively on a native resolution image. A native resolution image (304) is transformed into a multi-scale representation (302), enabling the Transformer's self-attention mechanism to capture information on both fine-grained detailed patches and coarse-grained global patches. Spatial embedding (316) is employed to map patch positions to a fixed grid, in which patch locations at each scale are hashed to the same grid. A separate scale embedding (318) is employed to distinguish patches coming from different scales in the multiscale representation. Self-attention (508) is performed to create a final image representation. In some instances, prior to performing self-attention, the system may prepend a learnable classification token (322) to the set of input tokens.

公开(公告)号：US20210374909A1

公开(公告)日：2021-12-02

申请号：US17263743

申请日：2019-08-06

Applicant: Google LLC

Inventor： Yi Hung Chen ,  Chia-Kai Liang ,  Bartlomiej Maciej Wronski ,  Peyman Milanfar ,  Ignacio Garcia Dorado

IPC: G06T3/40 ,  H04N5/232

Abstract: The present disclosure describes systems and techniques directed to optical image stabilization movement to create a super-resolution image of a scene. The systems and techniques include a user device (102) introducing (502), through an optical image stabilization system (114), movement to one or more components of a camera system (112) of the user device (102). The user device (102) then captures (504) respective and multiple frames (306) of an image of a scene, where the respective and multiple frames (306) of the image of the scene have respective, sub-pixel offsets of the image of the scene across the multiple frames (306) as a result of the introduced movement to the one or more components of the camera system (112). The user device (102) performs (506), based on the respective, sub-pixel offsets of the image of the scene across the respective, multiple frames (306), super-resolution computations and creates (508) the super-resolution image of the scene based on the super-resolution computations.

公开(公告)号：US11178430B2

公开(公告)日：2021-11-16

申请号：US16210900

申请日：2018-12-05

Applicant: Google LLC

Inventor： Peyman Milanfar ,  Feng Yang ,  Sungjoon Choi

IPC: H04N19/625 ,  H04N19/124 ,  H04N19/184

Abstract: Methods are provided for sharpening or otherwise modifying compressed images without decompressing and re-encoding the images. An overall image quality is determined based on the source of the compressed image, the quantization table of the compressed image, or some other factor(s), and a set of scaling factors corresponding to the image quality is selected. The selected scaling factors are then applied to corresponding quantization factors of the image's quantization table or other parameters of the compressed image that describe the image contents of the compressed image. The scaling factors of a given set of scaling factors can be determined by a machine learning process that involves training the scaling factors based on training images determined by decompressing and then sharpening or otherwise modifying a source set of compressed images. These methods can provide improvements with respect to encoded image size and computational cost of the image modification method.