公开(公告)号：US20230154111A1

公开(公告)日：2023-05-18

申请号：US17987586

申请日：2022-11-15

Applicant: Samsung Electronics Co., Ltd.

Inventor： Taras Andreevich KHAKHULIN ,  Vanessa Valerievna SKLYAROVA ,  Victor Sergeevich LEMPITSKY ,  Egor Olegovich ZAKHAROV

IPC: G06T17/20 ,  G06T9/00 ,  G06T7/70 ,  G06V10/74 ,  G06V10/82 ,  G06V40/16

CPC classification number: G06T17/205 ,  G06T9/00 ,  G06T7/70 ,  G06V10/761 ,  G06V10/82 ,  G06V40/174 ,  G06T2207/30201

Abstract: A method of three-dimensional reconstruction of human heads using a single photo in the form of polygonal mesh, with animation and realistic rendering capabilities for novel head poses is provided. The method includes encoding, by using a first convolutional neural network, a single source image into a neural texture; estimating, by a pre-trained detailed expression capture and animation (DECA) system, a face shape, a facial expression, and a head pose by using the single source image and a target image, and providing an initial mesh; providing a predicted mesh of a head mesh based on the initial mesh and the neural texture; rendering a human image by using the predicted mesh.

公开(公告)号：US20220270304A1

公开(公告)日：2022-08-25

申请号：US17741959

申请日：2022-05-11

Applicant: Samsung Electronics Co., Ltd.

Inventor： Gleb Mikhailovich STERKIN ,  Ivan Aleksandrovich ANOKHIN ,  Taras Andreevich KHAKHULIN ,  Aleksei Vladislavovich KHARLAMOV ,  Denis Mikhailovich KORZHENKOV ,  Victor Sergeevich LEMPITSKY ,  Sergey Igorevich NIKOLENKO ,  Aleksei Sergeevich SILVESTROV ,  Pavel Ilich SOLOVEV

IPC: G06T11/00 ,  G06T7/10 ,  G06T3/40

Abstract: The disclosure relates to a field of plausible timelapse image(s) generation from a single image. A method of generating one or more images of a plausible dayscale timelapse sequence based on a content image using a trained generative neural network and a trained merging neural network is provided. The method includes receiving the content image and one of one or more predefined styles respectively corresponding to times of day to be applied to the content image or style images having styles to be applied to the content image, slicing the content image into n image crops, applying the trained generative neural network with each style to n image crops to obtain n image crops re-stylized according to each style, and merging the re-stylized n image crops for each style with the trained merging neural network to obtain images of a plausible dayscale timelapse sequence for the content image.

公开(公告)号：US20200302184A1

公开(公告)日：2020-09-24

申请号：US16823752

申请日：2020-03-19

Applicant: Samsung Electronics Co., Ltd.

Inventor： Victor Sergeevich LEMPITSKY ,  Aliaksandra Petrovna SHYSHEYA ,  Egor Olegovich ZAKHAROV ,  Egor Andreevich BURKOV

IPC: G06K9/00 ,  G06N3/08

Abstract: An electronic device and a controlling method thereof are provided. A controlling method of an electronic device according to the disclosure includes: performing first learning for a neural network model for acquiring a video sequence including a talking head of a random user based on a plurality of learning video sequences including talking heads of a plurality of users, performing second learning for fine-tuning the neural network model based on at least one image including a talking head of a first user different from the plurality of users and first landmark information included in the at least one image, and acquiring a first video sequence including the talking head of the first user based on the at least one image and pre-stored second landmark information using the neural network model for which the first learning and the second learning were performed.

公开(公告)号：US20230169349A1

公开(公告)日：2023-06-01

申请号：US18102161

申请日：2023-01-27

Applicant: Samsung Electronics Co., Ltd.

Inventor： Victor Sergeevich LEMPITSKY ,  Aliaksandra Petrovna SHYSHEYA ,  Egor Olegovich ZAKHAROV ,  Egor Andreevich BURKOV

IPC: G06N3/088 ,  G06N3/08 ,  G06V20/40 ,  G06V40/16 ,  G06F16/70 ,  G06V40/00 ,  G06V10/764 ,  G06V10/774 ,  G06V10/82 ,  G06V40/20

CPC classification number: G06N3/088 ,  G06N3/08 ,  G06V20/46 ,  G06V20/41 ,  G06V40/172 ,  G06F16/70 ,  G06V40/00 ,  G06V40/168 ,  G06V40/169 ,  G06V40/179 ,  G06V10/764 ,  G06V10/7753 ,  G06V10/82 ,  G06V40/20 ,  G06T2207/10016 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/30201

Abstract: An electronic device and a controlling method thereof are provided. A controlling method of an electronic device according to the disclosure includes: performing first learning for a neural network model for acquiring a video sequence including a talking head of a random user based on a plurality of learning video sequences including talking heads of a plurality of users, performing second learning for fine-tuning the neural network model based on at least one image including a talking head of a first user different from the plurality of users and first landmark information included in the at least one image, and acquiring a first video sequence including the talking head of the first user based on the at least one image and pre-stored second landmark information using the neural network model for which the first learning and the second learning were performed.

公开(公告)号：US20220207646A1

公开(公告)日：2022-06-30

申请号：US17697436

申请日：2022-03-17

Applicant: Samsung Electronics Co., Ltd.

Inventor： Ivan Aleksandrovich ANOKHIN ,  Kirill Vladislavovich DEMOCHKIN ,  Taras Andreevich KHAKHULIN ,  Gleb Mikhailovich STERKIN ,  Victor Sergeevich LEMPITSKY ,  Denis Mikhailovich KORZHENKOV

IPC: G06T3/40 ,  G06N3/04

Abstract: The disclosure relates to multi-layer perceptron architecture, that may be used for image generation. A new architecture for image generators, where the color value at each pixel is computed independently given the value of a random latent vector and the coordinate of that pixel is provided. No spatial convolutions or similar operations that propagate information across pixels are involved during the synthesis.

公开(公告)号：US20240096011A1

公开(公告)日：2024-03-21

申请号：US18513716

申请日：2023-11-20

Applicant: SAMSUNG ELECTRONICS CO., LTD.

Inventor： Artem Mikhailovich SEVASTOPOLSKIY ,  Victor Sergeevich LEMPITSKY

IPC: G06T15/60 ,  G06F18/214 ,  G06N3/084 ,  G06T7/194 ,  G06T15/04 ,  G06T15/10 ,  G06T17/20

CPC classification number: G06T15/60 ,  G06F18/2148 ,  G06N3/084 ,  G06T7/194 ,  G06T15/04 ,  G06T15/10 ,  G06T17/20 ,  G06T2207/10016 ,  G06T2207/10152 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/30201 ,  G06T2207/30244 ,  G06T2215/12

Abstract: The disclosure provides a method for generating relightable 3D portrait using a deep neural network and a computing device implementing the method. A possibility of obtaining, in real time and on computing devices having limited processing resources, realistically relighted 3D portraits having quality higher or at least comparable to quality achieved by prior art solutions, but without utilizing complex and costly equipment is provided. A method for rendering a relighted 3D portrait of a person, the method including: receiving an input defining a camera viewpoint and lighting conditions, rasterizing latent descriptors of a 3D point cloud at different resolutions based on the camera viewpoint to obtain rasterized images, wherein the 3D point cloud is generated based on a sequence of images captured by a camera with a blinking flash while moving the camera at least partly around an upper body, the sequence of images comprising a set of flash images and a set of no-flash images, processing the rasterized images with a deep neural network to predict albedo, normals, environmental shadow maps, and segmentation mask for the received camera viewpoint, and fusing the predicted albedo, normals, environmental shadow maps, and segmentation mask into the relighted 3D portrait based on the lighting conditions.

公开(公告)号：US20230126829A1

公开(公告)日：2023-04-27

申请号：US18086328

申请日：2022-12-21

Applicant: SAMSUNG ELECTRONICS CO., LTD.

Inventor： Artur Andreevich GRIGOREV ,  Victor Sergeevich LEMPITSKY ,  llya Dmitrievich ZAKHARKIN ,  Kirill Yevgenevich MAZUR

IPC: G06T19/20 ,  G06T17/20 ,  G06T7/70 ,  G06T7/50

Abstract: Provided are virtual try-on applications, telepresence applications, relating to modeling realistic clothing worn by humans and realistic modeling of humans in three-dimension (3D). Proposed is a hardware comprising software products that perform method for imaging clothes on a person, that is adapted to the body pose and the body shape, based on point cloud draping model, the method including using of point cloud and a neural network that synthesizes such point clouds to capture/model the geometry of clothing outfits, and using of point based differentiable neural rendering to capture the appearance of clothing outfits.

公开(公告)号：US20230123532A1

公开(公告)日：2023-04-20

申请号：US18083354

申请日：2022-12-16

Applicant: SAMSUNG ELECTRONICS CO., LTD.

Inventor： Gleb STERKIN ,  Pavel Ilyich SOLOVEV ,  Denis Mikhaylovich KORZHENKOV ,  Victor Sergeevich LEMPITSKY ,  Taras Andreevich KHAKHULIN

IPC: G06T15/50 ,  G06T15/08 ,  G06T15/10 ,  G06T7/90 ,  G06T7/50 ,  G06T17/20

Abstract: The present disclosure relates to the field of artificial intelligence (AI) and neural rendering, and particularly to a method of generating a multi-layer representation of a scene using neural networks trained in an end-to-end fashion and to a computing device implementing the method. The method of generating a multi-layer representation of a scene includes: obtaining a pair of images of the scene, the pair of the images comprising a reference image and a source image; performing a reprojection operation on the pair of images to generate a plane-sweep volume; predicting, using a geometry network, a layered structure of the scene based on the plane-sweep volume; and estimating, using a coloring network, color values and opacity values for the predicted layered structure of the scene to obtain the multi-layer representation of the scene; wherein the geometry network and the coloring network are trained in end-to-end manner.

公开(公告)号：US20220157014A1

公开(公告)日：2022-05-19

申请号：US17457078

申请日：2021-12-01

Applicant: SAMSUNG ELECTRONICS CO., LTD.

Inventor： Artem Mikhailovich SEVASTOPOLSKIY ,  Victor Sergeevich LEMPITSKY

IPC: G06T15/60 ,  G06T7/194 ,  G06K9/62 ,  G06T15/04 ,  G06T17/20 ,  G06T15/10 ,  G06N3/08

Abstract: The disclosure provides a method for generating relightable 3D portrait using a deep neural network and a computing device implementing the method. A possibility of obtaining, in real time and on computing devices having limited processing resources, realistically relighted 3D portraits having quality higher or at least comparable to quality achieved by prior art solutions, but without utilizing complex and costly equipment is provided. A method for rendering a relighted 3D portrait of a person, the method including: receiving an input defining a camera viewpoint and lighting conditions, rasterizing latent descriptors of a 3D point cloud at different resolutions based on the camera viewpoint to obtain rasterized images, wherein the 3D point cloud is generated based on a sequence of images captured by a camera with a blinking flash while moving the camera at least partly around an upper body, the sequence of images comprising a set of flash images and a set of no-flash images, processing the rasterized images with a deep neural network to predict albedo, normals, environmental shadow maps, and segmentation mask for the received camera viewpoint, and fusing the predicted albedo, normals, environmental shadow maps, and segmentation mask into the relighted 3D portrait based on the lighting conditions.

公开(公告)号：US20210334935A1

公开(公告)日：2021-10-28

申请号：US17282214

申请日：2019-11-07

Applicant: Samsung Electronics Co., Ltd.

Inventor： Artur Andreevich GRIGORIEV ,  Victor Sergeevich LEMPITSKY ,  Artem Mikhailovich SEVASTOPOLSKY ,  Alexander Timurovich VAKHITOV

IPC: G06T3/00 ,  G06T5/00 ,  G06N3/08 ,  G06N3/04

Abstract: The invention relates to image processing and, in particular, to image resynthesis for synthesizing new views of a person or an object based on an input image, to resolve tasks such as predicting views of a person or an object from new viewpoints and in new poses. Technical result consists in improved accuracy of image resynthesis based on at least one input image. An image resynthesis system, a system for training a gap filling module to be used in the image resynthesis system, an image resynthesis method, a computer program product and a computer-readable medium are provided. The image resynthesis system comprises a source image input module, a forward warping module configured to predict, for each source image pixel, a corresponding position in a target image, the forward warping module being configured to predict a forward warping field which is aligned with the source image, and a gap filling module configured to fill in the gaps resulting from the application of the forward warping module. The image resynthesis method comprises the steps of: inputting a source image, predicting, for each source image pixel, a corresponding position in a target image, wherein a forward warping field which is aligned with the source image is predicted, predicting a binary mask of gaps which result from the forward warping, filling in the gaps based on said binary mask of gaps by generating a texture image by means of predicting a pair of coordinates in the source image for each pixel in the texture image, and mapping the whole texture back to a new pose using backward warping.