公开(公告)号：US12169626B2

公开(公告)日：2024-12-17

申请号：US18116003

申请日：2023-03-01

Applicant: Snap Inc.

Inventor： Aleksandr Zakharov ,  Sergey Smetanin ,  Arnab Ghosh ,  Pavel Savchenkov

IPC: G06F3/048 ,  G06F3/04845 ,  G06T11/00 ,  G06F40/12

Abstract: Examples disclosed herein describe prompt modification techniques for automated image generation. An image generation request comprising a base prompt is received from a user device. A plurality of prompt modifiers is identified. A processor-implemented scoring engine determines, for each prompt modifier, a modifier score. The modifier score for each prompt modifier is associated with the base prompt. One or more of the prompt modifiers are automatically selected based on the modifier scores. A modified prompt is generated. The modified prompt is based on the base prompt and the one or more selected prompt modifiers. The modified prompt is provided as input to an automated image generator to generate an image, and the image is caused to be presented on the user device.

公开(公告)号：US12125147B2

公开(公告)日：2024-10-22

申请号：US17663176

申请日：2022-05-12

Applicant: Snap Inc.

Inventor： Pavel Savchenkov ,  Yurii Volkov ,  Jeremy Baker Voss

IPC: G06T19/00 ,  G06N20/00 ,  G06T13/40 ,  H04L51/046

CPC classification number: G06T19/006 ,  G06N20/00 ,  G06T13/40 ,  H04L51/046 ,  G06T2200/24

Abstract: A methodology for training a machine learning model to generate color-neutral input face images is described. For each training face image from a training dataset that is used for training the model, the training system generates an input face image, which has the color and lighting of a randomly selected image from the set of color source images, and which has facial features and expression of a face object from the training face image. Because, during training, the machine learning model is “confused” by changing the color and lighting of a training face image to a randomly selected different color and lighting, the trained machine learning model generates a color neutral embedding representing facial features from the training face image.

公开(公告)号：US20240265498A1

公开(公告)日：2024-08-08

申请号：US18164458

申请日：2023-02-03

Applicant: Snap inc.

Inventor： Aleksandr Belskikh ,  Georgii Grigorev ,  Pavel Savchenkov

IPC: G06T5/50 ,  G06T3/40 ,  G06T5/00

CPC classification number: G06T5/50 ,  G06T3/4046 ,  G06T5/70 ,  G06T2200/24 ,  G06T2207/20084 ,  G06T2207/20221 ,  G06T2207/30201

Abstract: The subject technology receives an input image and a segmentation mask of the input image. The subject technology obtains reconstructed noise of the input image using the input image and the segmentation mask. The subject technology determines a first set of features by performing a first portion of a forward pass of the reconstructed noise through a decoder. The subject technology determines a second set of features by processing the input image for stable diffusion using an image to image (IMG2IMG) model. The subject technology generates a third set of features based on combining, using the segmentation mask, the first set of features and the second set of features with the reconstructed noise. The subject technology generates an output image by performing a remaining portion of the forward pass of the third set of features through the decoder.

公开(公告)号：US20240221259A1

公开(公告)日：2024-07-04

申请号：US18149007

申请日：2022-12-30

Applicant: Snap Inc.

Inventor： Aleksandr Belskikh ,  Menglei Chai ,  Antoine Chassang ,  Anna Kovalenko ,  Pavel Savchenkov

IPC: G06T13/40 ,  G06N3/094 ,  G06T19/00

CPC classification number: G06T13/40 ,  G06N3/094 ,  G06T19/006

Abstract: The subject technology generates a first image of a face using a GAN model. The subject technology applies 3D virtual hair on the first image to generate a second image with 3D virtual hair. The subject technology projects the second image with 3D virtual hair into a GAN latent space to generate a third image with realistic virtual hair. The subject technology performs a blend of the realistic virtual hair with the first image of the face to generate a new image with new realistic hair that corresponds to the 3D virtual hair. The subject technology trains a neural network that receives the second image with the 3D virtual hair and provides an output image with realistic virtual hair. The subject technology generates using the trained neural network, a particular output image with realistic hair based on a particular input image with 3D virtual hair.

公开(公告)号：US20230351998A1

公开(公告)日：2023-11-02

申请号：US18218028

申请日：2023-07-04

Applicant: Snap Inc.

Inventor： Pavel Savchenkov ,  Maxim Lukin ,  Aleksandr Mashrabov

IPC: G10L13/00 ,  G10L13/08 ,  G06T13/40 ,  G06V40/16 ,  G06V10/764 ,  G06V10/82

CPC classification number: G10L13/00 ,  G10L13/08 ,  G06T13/40 ,  G06V40/171 ,  G06V10/764 ,  G06V10/82

Abstract: Systems and methods for text and audio-based real-time face reenactment are provided. An example method includes receiving an input text and a target image, where the target image including a target face, generating, based on the input text, a sequence of sets of acoustic features corresponding to the input text, generating, based on the sequence of sets of acoustic features, a sequence of sets of mouth key points, generating, based on the sequence of sets of mouth key points, a sequence of sets of facial key points, determining, based on the set sequence of sets of facial key points, a sequence of deformations of the target face, and applying the sequence of deformations to the target image, thereby generating a sequence of frames of an output video.

公开(公告)号：US20220392133A1

公开(公告)日：2022-12-08

申请号：US17881947

申请日：2022-08-05

Applicant: Snap Inc.

Inventor： Yurii Volkov ,  Pavel Savchenkov ,  Nikolai Smirnov ,  Aleksandr Mashrabov

IPC: G06T13/40 ,  G06T7/70 ,  G06V40/16

Abstract: Provided are systems and methods for realistic head turns and face animation synthesis. An example method includes receiving a source frame of a source video, where the source frame includes a head and a face of a source actor, generating source pose parameters corresponding to a pose of the head and a facial expression of the source actor; receiving a target image including a target head and a target face of a target person, determining target identity information associated with the target head and the target face of the target person, replacing source identity information in the source pose parameters with the target identity information to obtain further source pose parameters, and generating an output frame of an output video that includes a modified image of the target face and the target head adopting the pose of the head and the facial expression of the source actor.

公开(公告)号：US20220319231A1

公开(公告)日：2022-10-06

申请号：US17709711

申请日：2022-03-31

Applicant: Snap Inc.

Inventor： Alexey Pankov ,  Pavel Savchenkov

IPC: G06V40/16 ,  G06T17/20 ,  G06T19/20 ,  G06T7/10 ,  G06V10/82 ,  G06T3/00

Abstract: The subject technology receives frames of a source media content, the frames of the source media content including representations of a head and a face of a source actor. The subject technology generates, based at least in part on the frames of the source media content, sets of source pose parameters. The subject technology receives at least one target image, the at least one target image including representations of a target head and a target face of a target entity. The subject technology provides the sets of source pose parameters to a neural network to determine facial landmarks for head turns and facial expressions. The subject technology generates, based at least in part on the sets of source pose parameters and the facial landmarks for head turns and facial expressions, an output media content. The subject technology provides augmented reality content based at least in part on the output media content for display on a computing device.

公开(公告)号：US20220270332A1

公开(公告)日：2022-08-25

申请号：US17663176

申请日：2022-05-12

Applicant: Snap Inc.

Inventor： Pavel Savchenkov ,  Yurii Volkov ,  Jeremy Baker Voss

IPC: G06T19/00 ,  G06N20/00 ,  G06T13/40 ,  H04L51/046

Abstract: A methodology for training a machine learning model to generate color-neutral input face images is described. For each training face image from a training dataset that is used for training the model, the training system generates an input face image, which has the color and lighting of a randomly selected image from the set of color source images, and which has facial features and expression of a face object from the training face image. Because, during training, the machine learning model is “confused” by changing the color and lighting of a training face image to a randomly selected different color and lighting, the trained machine learning model generates a color neutral embedding representing facial features from the training face image.

公开(公告)号：US11335069B1

公开(公告)日：2022-05-17

申请号：US17107410

申请日：2020-11-30

Applicant: Snap Inc.

Inventor： Pavel Savchenkov ,  Yurii Volkov ,  Jeremy Baker Voss

IPC: G06T19/00 ,  G06T13/40 ,  G06N20/00 ,  H04L51/046

Abstract: In some embodiments, users' experience of engaging with augmented reality technology is enhanced by providing a process, referred to as face animation synthesis, that replaces an actor's face in the frames of a video with a user's face from the user's portrait image. The resulting face in the frames of the video retains the facial expressions, as well as color and lighting, of the actor's face but, at the same time, has the likeness of the user's face. An example face animation synthesis experience can be made available to uses of a messaging system by providing a face animation synthesis augmented reality component.

公开(公告)号：US20210327404A1

公开(公告)日：2021-10-21

申请号：US17355834

申请日：2021-06-23

Applicant: Snap Inc.

Inventor： Pavel Savchenkov ,  Maxim Lukin ,  Aleksandr Mashrabov

IPC: G10L13/00 ,  G06K9/00 ,  G10L13/08 ,  G06T13/40

Abstract: Provided are systems and methods for text and audio-based real-time face reenactment. An example method includes receiving an input text and a target image, the target image including a target face; generating, based on the input text, a sequence of sets of acoustic features representing the input text; generating, based on the sequence of sets of acoustic features, a sequence of sets of mouth key points; generating, based on the sequence of sets of mouth key points, a sequence of sets of facial key points; generating, by the computing device and based on the sequence of sets of the facial key points and the target image, a sequence of frames; and generating, based on the sequence of frames, an output video. Each of the frames includes the target face modified based on at least one set of mouth key points of the sequence of sets of mouth key points.