公开(公告)号：US20230141734A1

公开(公告)日：2023-05-11

申请号：US17520249

申请日：2021-11-05

Applicant: Adobe Inc.

Inventor： Yuqian Zhou ,  Connelly Barnes ,  Sohrab Amirghodsi ,  Elya Shechtman

IPC: G06T5/00 ,  G06T7/11 ,  G06T11/00 ,  G06K9/62 ,  G06N3/02

CPC classification number: G06T5/005 ,  G06T7/11 ,  G06T11/00 ,  G06K9/6215 ,  G06N3/02 ,  G06T2207/20084

Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately generating inpainted digital images utilizing a guided inpainting model guided by both plane panoptic segmentation and plane grouping. For example, the disclosed systems utilize a guided inpainting model to fill holes of missing pixels of a digital image as informed or guided by an appearance guide and a geometric guide. Specifically, the disclosed systems generate an appearance guide utilizing plane panoptic segmentation and generate a geometric guide by grouping plane panoptic segments. In some embodiments, the disclosed systems generate a modified digital image by implementing an inpainting model guided by both the appearance guide (e.g., a plane panoptic segmentation map) and the geometric guide (e.g., a plane grouping map).

公开(公告)号：US11538140B2

公开(公告)日：2022-12-27

申请号：US17098055

申请日：2020-11-13

Applicant: ADOBE INC.

Inventor： Yuqian Zhou ,  Elya Shechtman ,  Connelly Stuart Barnes ,  Sohrab Amirghodsi

IPC: G06T5/00 ,  G06T5/50 ,  G06T3/00 ,  G06N3/08

Abstract: Various disclosed embodiments are directed to inpainting one or more portions of a target image based on merging (or selecting) one or more portions of a warped image with (or from) one or more portions of an inpainting candidate (e.g., via a learning model). This, among other functionality described herein, resolves the inaccuracies of existing image inpainting technologies.

公开(公告)号：US20210312599A1

公开(公告)日：2021-10-07

申请号：US17350136

申请日：2021-06-17

Applicant: ADOBE INC.

Inventor： Sohrab Amirghodsi ,  Elya Shechtman ,  Derek Novo

IPC: G06T5/00 ,  G06T7/194 ,  G06T7/90 ,  G06T7/70

Abstract: Embodiments of the present invention provide systems, methods, and computer storage media for automatically synthesizing a content-aware sampling region for a hole-filling algorithm such as content-aware fill. Given a source image and a hole (or other target region to fill), a sampling region can be synthesized by identifying a band of pixels surrounding the hole, clustering these pixels based on one or more characteristics (e.g., color, x/y coordinates, depth, focus, etc.), passing each of the resulting clusters as foreground pixels to a segmentation algorithm, and unioning the resulting pixels to form the sampling region. The sampling region can be stored in a constraint mask and passed to a hole-filling algorithm such as content-aware fill to synthesize a fill for the hole (or other target region) from patches sampled from the synthesized sampling region.

公开(公告)号：US11042969B2

公开(公告)日：2021-06-22

申请号：US16420782

申请日：2019-05-23

Applicant: ADOBE INC.

Inventor： Sohrab Amirghodsi ,  Elya Shechtman ,  Derek Novo

IPC: G06K9/00 ,  G06T5/00 ,  G06T7/194 ,  G06T7/90 ,  G06T7/70 ,  G06T5/20

Abstract: Embodiments of the present invention provide systems, methods, and computer storage media for automatically synthesizing a content-aware sampling region for a hole-filling algorithm such as content-aware fill. Given a source image and a hole (or other target region to fill), a sampling region can be synthesized by identifying a band of pixels surrounding the hole, clustering these pixels based on one or more characteristics (e.g., color, x/y coordinates, depth, focus, etc.), passing each of the resulting clusters as foreground pixels to a segmentation algorithm, and unioning the resulting pixels to form the sampling region. The sampling region can be stored in a constraint mask and passed to a hole-filling algorithm such as content-aware fill to synthesize a fill for the hole (or other target region) from patches sampled from the synthesized sampling region.

公开(公告)号：US20210142463A1

公开(公告)日：2021-05-13

申请号：US16678132

申请日：2019-11-08

Applicant: Adobe Inc.

Inventor： Sohrab Amirghodsi ,  Aliakbar Darabi ,  Elya Shechtman

IPC: G06T7/00 ,  G06T3/40 ,  G06T9/00 ,  G06K9/62

Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media for generating modified digital images by utilizing a patch match algorithm to generate nearest neighbor fields for a second digital image based on a nearest neighbor field associated with a first digital image. For example, the disclosed systems can identify a nearest neighbor field associated with a first digital image of a first resolution. Based on the nearest neighbor field of the first digital image, the disclosed systems can utilize a patch match algorithm to generate a nearest neighbor field for a second digital image of a second resolution larger than the first resolution. The disclosed systems can further generate a modified digital image by filling a target region of the second digital image utilizing the generated nearest neighbor field.

公开(公告)号：US20210056668A1

公开(公告)日：2021-02-25

申请号：US16548498

申请日：2019-08-22

Applicant: Adobe Inc.

Inventor： Connelly Barnes ,  Sohrab Amirghodsi ,  Elya Shechtman

IPC: G06T5/00 ,  G06T5/50 ,  G06T7/33

Abstract: Techniques are disclosed for filling or otherwise replacing a target region of a primary image with a corresponding region of an auxiliary image. The filling or replacing can be done with an overlay (no subtractive process need be run on the primary image). Because the primary and auxiliary images may not be aligned, both geometric and photometric transformations are applied to the primary and/or auxiliary images. For instance, a geometric transformation of the auxiliary image is performed, to better align features of the auxiliary image with corresponding features of the primary image. Also, a photometric transformation of the auxiliary image is performed, to better match color of one or more pixels of the auxiliary image with color of corresponding one or more pixels of the primary image. The corresponding region of the transformed auxiliary image is then copied and overlaid on the target region of the primary image.

公开(公告)号：US20190050961A1

公开(公告)日：2019-02-14

申请号：US16160855

申请日：2018-10-15

Applicant: ADOBE INC.

Inventor： Sohrab Amirghodsi ,  Aliakbar Darabi ,  Elya Shechtman

IPC: G06T3/40 ,  G06T7/73

Abstract: Embodiments of the present invention provide systems, methods, and computer storage media directed at image synthesis utilizing sampling of patch correspondence information between iterations at different scales. A patch synthesis technique can be performed to synthesize a target region at a first image scale based on portions of a source region that are identified by the patch synthesis technique. The image can then be sampled to generate an image at a second image scale. The sampling can include generating patch correspondence information for the image at the second image scale. Invalid patch assignments in the patch correspondence information at the second image scale can then be identified, and valid patches can be assigned to the pixels having invalid patch assignments. Other embodiments may be described and/or claimed.

公开(公告)号：US12299844B2

公开(公告)日：2025-05-13

申请号：US18440248

申请日：2024-02-13

Applicant: Adobe Inc.

Inventor： He Zhang ,  Yifan Jiang ,  Yilin Wang ,  Jianming Zhang ,  Kalyan Sunkavalli ,  Sarah Kong ,  Su Chen ,  Sohrab Amirghodsi ,  Zhe Lin

IPC: G06T3/08 ,  G06N3/04 ,  G06N3/08 ,  G06T5/50 ,  G06T7/194 ,  G06T11/00 ,  G06T11/60

Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately, efficiently, and flexibly generating harmonized digital images utilizing a self-supervised image harmonization neural network. In particular, the disclosed systems can implement, and learn parameters for, a self-supervised image harmonization neural network to extract content from one digital image (disentangled from its appearance) and appearance from another from another digital image (disentangled from its content). For example, the disclosed systems can utilize a dual data augmentation method to generate diverse triplets for parameter learning (including input digital images, reference digital images, and pseudo ground truth digital images), via cropping a digital image with perturbations using three-dimensional color lookup tables (“LUTs”). Additionally, the disclosed systems can utilize the self-supervised image harmonization neural network to generate harmonized digital images that depict content from one digital image having the appearance of another digital image.

公开(公告)号：US12190484B2

公开(公告)日：2025-01-07

申请号：US17202019

申请日：2021-03-15

Applicant: Adobe Inc.

Inventor： Sohrab Amirghodsi ,  Lingzhi Zhang ,  Zhe Lin ,  Connelly Barnes ,  Elya Shechtman

IPC: G06T5/77 ,  G06N3/08 ,  G06T3/4053 ,  G06T7/11 ,  G06T7/50

Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately, efficiently, and flexibly generating modified digital images utilizing a guided inpainting approach that implements a patch match model informed by a deep visual guide. In particular, the disclosed systems can utilize a visual guide algorithm to automatically generate guidance maps to help identify replacement pixels for inpainting regions of digital images utilizing a patch match model. For example, the disclosed systems can generate guidance maps in the form of structure maps, depth maps, or segmentation maps that respectively indicate the structure, depth, or segmentation of different portions of digital images. Additionally, the disclosed systems can implement a patch match model to identify replacement pixels for filling regions of digital images according to the structure, depth, and/or segmentation of the digital images.

公开(公告)号：US12165295B2

公开(公告)日：2024-12-10

申请号：US17661985

申请日：2022-05-04

Applicant: Adobe Inc.

Inventor： Haitian Zheng ,  Zhe Lin ,  Jingwan Lu ,  Scott Cohen ,  Elya Shechtman ,  Connelly Barnes ,  Jianming Zhang ,  Ning Xu ,  Sohrab Amirghodsi

IPC: G06T5/77 ,  G06T3/4046 ,  G06V10/40

Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media that generate inpainted digital images utilizing a cascaded modulation inpainting neural network. For example, the disclosed systems utilize a cascaded modulation inpainting neural network that includes cascaded modulation decoder layers. For example, in one or more decoder layers, the disclosed systems start with global code modulation that captures the global-range image structures followed by an additional modulation that refines the global predictions. Accordingly, in one or more implementations, the image inpainting system provides a mechanism to correct distorted local details. Furthermore, in one or more implementations, the image inpainting system leverages fast Fourier convolutions block within different resolution layers of the encoder architecture to expand the receptive field of the encoder and to allow the network encoder to better capture global structure.