公开(公告)号：US20240161240A1

公开(公告)日：2024-05-16

申请号：US18053027

申请日：2022-11-07

Applicant: Adobe Inc.

Inventor： He Zhang ,  Hyun Joon Jung

IPC: G06T5/50 ,  G06T7/11 ,  G06T7/194 ,  G06V10/26 ,  G06V10/42 ,  G06V10/44 ,  G06V10/82

CPC classification number: G06T5/50 ,  G06T7/11 ,  G06T7/194 ,  G06V10/267 ,  G06V10/42 ,  G06V10/44 ,  G06V10/82 ,  G06T2200/24 ,  G06T2207/20084 ,  G06T2207/20092 ,  G06T2207/20132 ,  G06T2207/20212

Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods that implement a multi-branch harmonization neural network architecture to harmonize composite images. For example, in one or more implementations, the semantic-guided transformer-based harmonization system uses a convolutional branch, a transformer branch, and a semantic branch to generate a harmonized composite image based on an input composite image and a corresponding segmentation mask. More particularly, the convolutional branch comprises a series of convolutional neural network layers followed by a style normalization layer to extract localized information from the input composite image. Further, the transformer branch comprises a series of transformer neural network layers to extract global information based on different resolutions of the input composite image. The semantic branch includes a visual neural network that generates semantic features that inform the harmonization of the composite images.

公开(公告)号：US20230298148A1

公开(公告)日：2023-09-21

申请号：US17655663

申请日：2022-03-21

Applicant: Adobe Inc.

Inventor： He Zhang ,  Jianming Zhang ,  Jose Ignacio Echevarria Vallespi ,  Kalyan Sunkavalli ,  Meredith Payne Stotzner ,  Yinglan Ma ,  Zhe Lin ,  Elya Shechtman ,  Frederick Mandia

IPC: G06T5/50 ,  G06T7/194 ,  G06T7/90 ,  G06T11/00

CPC classification number: G06T5/50 ,  G06T7/194 ,  G06T7/90 ,  G06T11/001 ,  G06T2207/20084 ,  G06T2207/20212 ,  G06T2200/24 ,  G06T2207/20092 ,  G06T2207/20016 ,  G06T2207/20081 ,  G06T2207/30168

Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods that implement a dual-branched neural network architecture to harmonize composite images. For example, in one or more implementations, the transformer-based harmonization system uses a convolutional branch and a transformer branch to generate a harmonized composite image based on an input composite image and a corresponding segmentation mask. More particularly, the convolutional branch comprises a series of convolutional neural network layers followed by a style normalization layer to extract localized information from the input composite image. Further, the transformer branch comprises a series of transformer neural network layers to extract global information based on different resolutions of the input composite image. Utilizing a decoder, the transformer-based harmonization system combines the local information and the global information from the corresponding convolutional branch and transformer branch to generate a harmonized composite image.

公开(公告)号：US11568544B2

公开(公告)日：2023-01-31

申请号：US17483280

申请日：2021-09-23

Applicant: Adobe Inc.

Inventor： Zhe Lin ,  Jianming Zhang ,  He Zhang ,  Federico Perazzi

IPC: G06K9/62 ,  G06T7/10 ,  G06N3/04 ,  G06N3/08 ,  G06T11/60

Abstract: The present disclosure relates to utilizing a neural network having a two-stream encoder architecture to accurately generate composite digital images that realistically portray a foreground object from one digital image against a scene from another digital image. For example, the disclosed systems can utilize a foreground encoder of the neural network to identify features from a foreground image and further utilize a background encoder to identify features from a background image. The disclosed systems can then utilize a decoder to fuse the features together and generate a composite digital image. The disclosed systems can train the neural network utilizing an easy-to-hard data augmentation scheme implemented via self-teaching. The disclosed systems can further incorporate the neural network within an end-to-end framework for automation of the image composition process.

公开(公告)号：US20220292654A1

公开(公告)日：2022-09-15

申请号：US17200338

申请日：2021-03-12

Applicant: Adobe Inc.

Inventor： He Zhang ,  Yifan Jiang ,  Yilin Wang ,  Jianming Zhang ,  Kalyan Sunkavalli ,  Sarah Kong ,  Su Chen ,  Sohrab Amirghodsi ,  Zhe Lin

IPC: G06T5/50 ,  G06T7/194 ,  G06T11/60 ,  G06T11/00 ,  G06N3/04 ,  G06N3/08

Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately, efficiently, and flexibly generating harmonized digital images utilizing a self-supervised image harmonization neural network. In particular, the disclosed systems can implement, and learn parameters for, a self-supervised image harmonization neural network to extract content from one digital image (disentangled from its appearance) and appearance from another from another digital image (disentangled from its content). For example, the disclosed systems can utilize a dual data augmentation method to generate diverse triplets for parameter learning (including input digital images, reference digital images, and pseudo ground truth digital images), via cropping a digital image with perturbations using three-dimensional color lookup tables (“LUTs”). Additionally, the disclosed systems can utilize the self-supervised image harmonization neural network to generate harmonized digital images that depict content from one digital image having the appearance of another digital image.

公开(公告)号：US12223623B2

公开(公告)日：2025-02-11

申请号：US18053027

申请日：2022-11-07

Applicant: Adobe Inc.

Inventor： He Zhang ,  Hyun Joon Jung

IPC: G06T5/50 ,  G06T7/11 ,  G06T7/194 ,  G06V10/26 ,  G06V10/42 ,  G06V10/44 ,  G06V10/82

Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods that implement a multi-branch harmonization neural network architecture to harmonize composite images. For example, in one or more implementations, the semantic-guided transformer-based harmonization system uses a convolutional branch, a transformer branch, and a semantic branch to generate a harmonized composite image based on an input composite image and a corresponding segmentation mask. More particularly, the convolutional branch comprises a series of convolutional neural network layers followed by a style normalization layer to extract localized information from the input composite image. Further, the transformer branch comprises a series of transformer neural network layers to extract global information based on different resolutions of the input composite image. The semantic branch includes a visual neural network that generates semantic features that inform the harmonization of the composite images.

公开(公告)号：US12169895B2

公开(公告)日：2024-12-17

申请号：US17502782

申请日：2021-10-15

Applicant: Adobe Inc.

Inventor： Yifan Liu ,  Jianming Zhang ,  He Zhang ,  Elya Shechtman ,  Zhe Lin

IPC: G06T15/60 ,  G06N3/045 ,  G06T7/194 ,  G06T7/70 ,  G06T11/00

Abstract: The present disclosure relates to systems, methods, and non-transitory computer-readable media that generate a height map for a digital object portrayed in a digital image and further utilizes the height map to generate a shadow for the digital object. Indeed, in one or more embodiments, the disclosed systems generate (e.g., utilizing a neural network) a height map that indicates the pixels heights for pixels of a digital object portrayed in a digital image. The disclosed systems utilize the pixel heights, along with lighting information for the digital image, to determine how the pixels of the digital image project to create a shadow for the digital object. Further, in some implementations, the disclosed systems utilize the determined shadow projections to generate (e.g., utilizing another neural network) a soft shadow for the digital object. Accordingly, in some cases, the disclosed systems modify the digital image to include the shadow.

公开(公告)号：US20240394889A1

公开(公告)日：2024-11-28

申请号：US18200908

申请日：2023-05-23

Applicant: Adobe Inc.

Inventor： He Zhang ,  Salil Tambe

IPC: G06T7/12 ,  G06F3/04845 ,  G06T5/50 ,  G06T7/13 ,  G06V10/56

Abstract: An image editing system accesses an input image displayed via a user interface and generates an instance-aware trimap for the input image by applying an instance-aware image segmentation model to input data including the input image and a segmented image defining a segment of the input image including a first set of pixel values. The trimap defines a modified segment using a second set of pixels different from the first set of pixels. Applying the model includes detecting boundaries of an object depicted in the input image. The second set of pixels is located within the boundaries of the object. Responsive to receiving a request via the user interface, the system generates a modified image by performing an editing operation on the input image including editing a portion of the second set of pixels of the modified segment of the trimap. The system transmits, for display, the modified image.

公开(公告)号：US20240185393A1

公开(公告)日：2024-06-06

申请号：US18440248

申请日：2024-02-13

Applicant: Adobe Inc.

Inventor： He Zhang ,  Yifan Jiang ,  Yilin Wang ,  Jianming Zhang ,  Kalyan Sunkavalli ,  Sarah Kong ,  Su Chen ,  Sohrab Amirghodsi ,  Zhe Lin

IPC: G06T5/50 ,  G06N3/04 ,  G06N3/08 ,  G06T7/194 ,  G06T11/00 ,  G06T11/60

CPC classification number: G06T5/50 ,  G06N3/04 ,  G06N3/08 ,  G06T7/194 ,  G06T11/001 ,  G06T11/60 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/20092 ,  G06T2207/20132 ,  G06T2207/20212

Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately, efficiently, and flexibly generating harmonized digital images utilizing a self-supervised image harmonization neural network. In particular, the disclosed systems can implement, and learn parameters for, a self-supervised image harmonization neural network to extract content from one digital image (disentangled from its appearance) and appearance from another from another digital image (disentangled from its content). For example, the disclosed systems can utilize a dual data augmentation method to generate diverse triplets for parameter learning (including input digital images, reference digital images, and pseudo ground truth digital images), via cropping a digital image with perturbations using three-dimensional color lookup tables (“LUTs”). Additionally, the disclosed systems can utilize the self-supervised image harmonization neural network to generate harmonized digital images that depict content from one digital image having the appearance of another digital image.

公开(公告)号：US11935217B2

公开(公告)日：2024-03-19

申请号：US17200338

申请日：2021-03-12

Applicant: Adobe Inc.

Inventor： He Zhang ,  Yifan Jiang ,  Yilin Wang ,  Jianming Zhang ,  Kalyan Sunkavalli ,  Sarah Kong ,  Su Chen ,  Sohrab Amirghodsi ,  Zhe Lin

IPC: G06T7/00 ,  G06N3/04 ,  G06N3/08 ,  G06T5/50 ,  G06T7/194 ,  G06T11/00 ,  G06T11/60

CPC classification number: G06T5/50 ,  G06N3/04 ,  G06N3/08 ,  G06T7/194 ,  G06T11/001 ,  G06T11/60 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/20092 ,  G06T2207/20132 ,  G06T2207/20212

Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately, efficiently, and flexibly generating harmonized digital images utilizing a self-supervised image harmonization neural network. In particular, the disclosed systems can implement, and learn parameters for, a self-supervised image harmonization neural network to extract content from one digital image (disentangled from its appearance) and appearance from another from another digital image (disentangled from its content). For example, the disclosed systems can utilize a dual data augmentation method to generate diverse triplets for parameter learning (including input digital images, reference digital images, and pseudo ground truth digital images), via cropping a digital image with perturbations using three-dimensional color lookup tables (“LUTs”). Additionally, the disclosed systems can utilize the self-supervised image harmonization neural network to generate harmonized digital images that depict content from one digital image having the appearance of another digital image.

公开(公告)号：US11875510B2

公开(公告)日：2024-01-16

申请号：US17200525

申请日：2021-03-12

Applicant: Adobe Inc.

Inventor： Yilin Wang ,  Chenglin Yang ,  Jianming Zhang ,  He Zhang ,  Zhe Lin

IPC: G06K9/00 ,  G06T7/11 ,  G06N3/08 ,  G06T3/40 ,  G06F18/213 ,  G06N3/044

CPC classification number: G06T7/11 ,  G06F18/213 ,  G06N3/044 ,  G06N3/08 ,  G06T3/4046 ,  G06T2207/20084

Abstract: The present disclosure relates to systems, methods, and non-transitory computer-readable media that utilizes a neural network having a hierarchy of hierarchical point-wise refining blocks to generate refined segmentation masks for high-resolution digital visual media items. For example, in one or more embodiments, the disclosed systems utilize a segmentation refinement neural network having an encoder and a recursive decoder to generate the refined segmentation masks. The recursive decoder includes a deconvolution branch for generating feature maps and a refinement branch for generating and refining segmentation masks. In particular, in some cases, the refinement branch includes a hierarchy of hierarchical point-wise refining blocks that recursively refine a segmentation mask generated for a digital visual media item. In some cases, the disclosed systems utilize a segmentation refinement neural network that includes a low-resolution network and a high-resolution network, each including an encoder and a recursive decoder, to generate the refined segmentation masks.