公开(公告)号：US12229880B2

公开(公告)日：2025-02-18

申请号：US18513716

申请日：2023-11-20

Applicant: SAMSUNG ELECTRONICS CO., LTD.

Inventor： Artem Mikhailovich Sevastopolskiy ,  Victor Sergeevich Lempitsky

IPC: G06T15/60 ,  G06F18/214 ,  G06N3/084 ,  G06T7/194 ,  G06T15/04 ,  G06T15/10 ,  G06T17/20

Abstract: The disclosure provides a method for generating relightable 3D portrait using a deep neural network and a computing device implementing the method. A possibility of obtaining, in real time and on computing devices having limited processing resources, realistically relighted 3D portraits having quality higher or at least comparable to quality achieved by prior art solutions, but without utilizing complex and costly equipment is provided. A method for rendering a relighted 3D portrait of a person, the method including: receiving an input defining a camera viewpoint and lighting conditions, rasterizing latent descriptors of a 3D point cloud at different resolutions based on the camera viewpoint to obtain rasterized images, wherein the 3D point cloud is generated based on a sequence of images captured by a camera with a blinking flash while moving the camera at least partly around an upper body, the sequence of images comprising a set of flash images and a set of no-flash images, processing the rasterized images with a deep neural network to predict albedo, normals, environmental shadow maps, and segmentation mask for the received camera viewpoint, and fusing the predicted albedo, normals, environmental shadow maps, and segmentation mask into the relighted 3D portrait based on the lighting conditions.

公开(公告)号：US12169900B2

公开(公告)日：2024-12-17

申请号：US17987586

申请日：2022-11-15

Applicant: SAMSUNG ELECTRONICS CO., LTD.

Inventor： Taras Andreevich Khakhulin ,  Vanessa Valerievna Sklyarova ,  Victor Sergeevich Lempitsky ,  Egor Olegovich Zakharov

IPC: G06T15/00 ,  G06T7/70 ,  G06T9/00 ,  G06T17/20 ,  G06V10/74 ,  G06V10/82 ,  G06V40/16

Abstract: A method of three-dimensional reconstruction of human heads using a single photo in the form of polygonal mesh, with animation and realistic rendering capabilities for novel head poses is provided. The method includes encoding, by using a first convolutional neural network, a single source image into a neural texture; estimating, by a pre-trained detailed expression capture and animation (DECA) system, a face shape, a facial expression, and a head pose by using the single source image and a target image, and providing an initial mesh; providing a predicted mesh of a head mesh based on the initial mesh and the neural texture; rendering a human image by using the predicted mesh.

公开(公告)号：US12039456B2

公开(公告)日：2024-07-16

申请号：US18102161

申请日：2023-01-27

Applicant: Samsung Electronics Co., Ltd.

Inventor： Victor Sergeevich Lempitsky ,  Aliaksandra Petrovna Shysheya ,  Egor Olegovich Zakharov ,  Egor Andreevich Burkov

IPC: G06V40/00 ,  G06F16/70 ,  G06N3/08 ,  G06N3/088 ,  G06V10/764 ,  G06V10/774 ,  G06V10/82 ,  G06V20/40 ,  G06V40/16 ,  G06V40/20

CPC classification number: G06N3/088 ,  G06F16/70 ,  G06N3/08 ,  G06V10/764 ,  G06V10/7753 ,  G06V10/82 ,  G06V20/41 ,  G06V20/46 ,  G06V40/00 ,  G06V40/168 ,  G06V40/169 ,  G06V40/172 ,  G06V40/179 ,  G06V40/20 ,  G06T2207/10016 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/30201

Abstract: An electronic device and a controlling method thereof are provided. A controlling method of an electronic device according to the disclosure includes: performing first learning for a neural network model for acquiring a video sequence including a talking head of a random user based on a plurality of learning video sequences including talking heads of a plurality of users, performing second learning for fine-tuning the neural network model based on at least one image including a talking head of a first user different from the plurality of users and first landmark information included in the at least one image, and acquiring a first video sequence including the talking head of the first user based on the at least one image and pre-stored second landmark information using the neural network model for which the first learning and the second learning were performed.

公开(公告)号：US11961205B2

公开(公告)日：2024-04-16

申请号：US17282214

申请日：2019-11-07

Applicant: Samsung Electronics Co., Ltd.

Inventor： Artur Andreevich Grigoriev ,  Victor Sergeevich Lempitsky ,  Artem Mikhailovich Sevastopolsky ,  Alexander Timurovich Vakhitov

IPC: G06K9/00 ,  G06N3/045 ,  G06N3/08 ,  G06T3/18 ,  G06T5/77

CPC classification number: G06T3/18 ,  G06N3/045 ,  G06N3/08 ,  G06T5/77 ,  G06T2207/20081 ,  G06T2207/20084

Abstract: An image resynthesis system, a system for training a gap filling module to be used in the image resynthesis system, an image resynthesis method, a computer program product, and a computer-readable medium are provided. The image resynthesis system comprises a source image input module, a forward warping module predicting, for each source image pixel of a source image, a corresponding position in a target image, and predicting a forward warping field which is aligned with the source image, and a gap filling module filling in gaps resulting from application of the forward warping module.

公开(公告)号：US12190440B2

公开(公告)日：2025-01-07

申请号：US18083354

申请日：2022-12-16

Applicant: SAMSUNG ELECTRONICS CO., LTD.

Inventor： Gleb Mikhaylovich Sterkin ,  Pavel Ilyich Solovev ,  Denis Mikhaylovich Korzhenkov ,  Victor Sergeevich Lempitsky ,  Taras Andreevich Khakhulin

IPC: G06T15/50 ,  G06T7/50 ,  G06T7/90 ,  G06T15/08 ,  G06T15/10 ,  G06T17/20

Abstract: The present disclosure relates to the field of artificial intelligence (AI) and neural rendering, and particularly to a method of generating a multi-layer representation of a scene using neural networks trained in an end-to-end fashion and to a computing device implementing the method. The method of generating a multi-layer representation of a scene includes: obtaining a pair of images of the scene, the pair of the images comprising a reference image and a source image; performing a reprojection operation on the pair of images to generate a plane-sweep volume; predicting, using a geometry network, a layered structure of the scene based on the plane-sweep volume; and estimating, using a coloring network, color values and opacity values for the predicted layered structure of the scene to obtain the multi-layer representation of the scene; wherein the geometry network and the coloring network are trained in end-to-end manner.

公开(公告)号：US11823349B2

公开(公告)日：2023-11-21

申请号：US17697436

申请日：2022-03-17

Applicant: Samsung Electronics Co., Ltd.

Inventor： Ivan Aleksandrovich Anokhin ,  Kirill Vladislavovich Demochkin ,  Taras Andreevich Khakhulin ,  Gleb Mikhailovich Sterkin ,  Victor Sergeevich Lempitsky ,  Denis Mikhailovich Korzhenkov

IPC: G06T3/40 ,  G06N3/04 ,  G06N3/08 ,  G06T11/60 ,  G06N3/048 ,  G06N3/045 ,  G06N3/047

CPC classification number: G06T3/4038 ,  G06N3/04 ,  G06N3/045 ,  G06N3/047 ,  G06N3/048 ,  G06N3/08 ,  G06T11/60

Abstract: The disclosure relates to multi-layer perceptron architecture, that may be used for image generation. A new architecture for image generators, where the color value at each pixel is computed independently given the value of a random latent vector and the coordinate of that pixel is provided. No spatial convolutions or similar operations that propagate information across pixels are involved during the synthesis.

公开(公告)号：US11823327B2

公开(公告)日：2023-11-21

申请号：US17457078

申请日：2021-12-01

Applicant: SAMSUNG ELECTRONICS CO., LTD.

Inventor： Artem Mikhailovich Sevastopolskiy ,  Victor Sergeevich Lempitsky

IPC: G06T15/60 ,  G06T15/04 ,  G06T17/20 ,  G06T15/10 ,  G06T7/194 ,  G06N3/084 ,  G06F18/214

CPC classification number: G06T15/60 ,  G06F18/2148 ,  G06N3/084 ,  G06T7/194 ,  G06T15/04 ,  G06T15/10 ,  G06T17/20 ,  G06T2207/10016 ,  G06T2207/10152 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/30201 ,  G06T2207/30244 ,  G06T2215/12

Abstract: The disclosure provides a method for generating relightable 3D portrait using a deep neural network and a computing device implementing the method. A possibility of obtaining, in real time and on computing devices having limited processing resources, realistically relighted 3D portraits having quality higher or at least comparable to quality achieved by prior art solutions, but without utilizing complex and costly equipment is provided. A method for rendering a relighted 3D portrait of a person, the method including: receiving an input defining a camera viewpoint and lighting conditions, rasterizing latent descriptors of a 3D point cloud at different resolutions based on the camera viewpoint to obtain rasterized images, wherein the 3D point cloud is generated based on a sequence of images captured by a camera with a blinking flash while moving the camera at least partly around an upper body, the sequence of images comprising a set of flash images and a set of no-flash images, processing the rasterized images with a deep neural network to predict albedo, normals, environmental shadow maps, and segmentation mask for the received camera viewpoint, and fusing the predicted albedo, normals, environmental shadow maps, and segmentation mask into the relighted 3D portrait based on the lighting conditions.

公开(公告)号：US11568645B2

公开(公告)日：2023-01-31

申请号：US16823752

申请日：2020-03-19

Applicant: Samsung Electronics Co., Ltd.

Inventor： Victor Sergeevich Lempitsky ,  Aliaksandra Petrovna Shysheya ,  Egor Olegovich Zakharov ,  Egor Andreevich Burkov

IPC: G06V40/00 ,  G06V20/40 ,  G06N3/08 ,  G06V40/16 ,  G06F16/70

Abstract: An electronic device and a controlling method thereof are provided. A controlling method of an electronic device according to the disclosure includes: performing first learning for a neural network model for acquiring a video sequence including a talking head of a random user based on a plurality of learning video sequences including talking heads of a plurality of users, performing second learning for fine-tuning the neural network model based on at least one image including a talking head of a first user different from the plurality of users and first landmark information included in the at least one image, and acquiring a first video sequence including the talking head of the first user based on the at least one image and pre-stored second landmark information using the neural network model for which the first learning and the second learning were performed.