公开(公告)号：US20250111592A1

公开(公告)日：2025-04-03

申请号：US18892186

申请日：2024-09-20

Applicant: NVIDIA Corporation

Inventor： Dejia Xu ,  Morteza Mardani ,  Jiaming Song ,  Sifei Liu ,  Ye Yuan ,  Arash Vahdat

IPC: G06T15/20 ,  G06V10/774 ,  G06V10/776 ,  G06V10/82

Abstract: Virtual reality and augmented reality bring increasing demand for 3D content creation. In an effort to automate the generation of 3D content, artificial intelligence-based processes have been developed. However, these processes are limited in terms of the quality of their output because they typically involve a model trained on limited 3D data thereby resulting in a model that does not generalize well to unseen objects, or a model trained on 2D data thereby resulting in a model that suffers from poor geometry due to ignorance of 3D information. The present disclosure jointly uses both 2D and 3D data to train a machine learning model to be able to generate 3D content from a single 2D image.

公开(公告)号：US12100113B2

公开(公告)日：2024-09-24

申请号：US17584213

申请日：2022-01-25

Applicant: NVIDIA Corporation

Inventor： Ye Yuan ,  Umar Iqbal ,  Pavlo Molchanov ,  Jan Kautz

IPC: G06T19/20 ,  G06T7/00 ,  G06T7/20

CPC classification number: G06T19/20 ,  G06T7/0002 ,  G06T7/20 ,  G06T2207/10016 ,  G06T2207/20084 ,  G06T2207/30241 ,  G06T2219/2016

Abstract: In order to determine accurate three-dimensional (3D) models for objects within a video, the objects are first identified and tracked within the video, and a pose and shape are estimated for these tracked objects. A translation and global orientation are removed from the tracked objects to determine local motion for the objects, and motion infilling is performed to fill in any missing portions for the object within the video. A global trajectory is then determined for the objects within the video, and the infilled motion and global trajectory are then used to determine infilled global motion for the object within the video. This enables the accurate depiction of each object as a 3D pose sequence for that model that accounts for occlusions and global factors within the video.

公开(公告)号：US20240169636A1

公开(公告)日：2024-05-23

申请号：US18317378

申请日：2023-05-15

Applicant: NVIDIA Corporation

Inventor： Ye Yuan ,  Jiaming Song ,  Umar Iqbal ,  Arash Vahdat ,  Jan Kautz

IPC: G06T13/40 ,  G06T5/00 ,  G06T13/80

CPC classification number: G06T13/40 ,  G06T5/002 ,  G06T13/80 ,  G06T2207/20081 ,  G06T2207/20084

Abstract: Systems and methods are disclosed that improve performance of synthesized motion generated by a diffusion neural network model. A physics-guided motion diffusion model incorporates physical constraints into the diffusion process to model the complex dynamics induced by forces and contact. Specifically, a physics-based motion projection module uses motion imitation in a physics simulator to project the denoised motion of a diffusion step to a physically plausible motion. The projected motion is further used in the next diffusion iteration to guide the denoising diffusion process. The use of physical constraints in the physics-guided motion diffusion model iteratively pulls the motion toward a physically-plausible space, reducing artifacts such as floating, foot sliding, and ground penetration.

公开(公告)号：US20240161377A1

公开(公告)日：2024-05-16

申请号：US18194116

申请日：2023-03-31

Applicant: NVIDIA Corporation

Inventor： Zhengyi Luo ,  Jason Peng ,  Sanja Fidler ,  Or Litany ,  Davis Winston Rempe ,  Ye Yuan

IPC: G06T13/40 ,  G06N3/006

CPC classification number: G06T13/40 ,  G06N3/006

Abstract: In various examples, systems and methods are disclosed relating to generating a simulated environment and update a machine learning model to move each of a plurality of human characters having a plurality of body shapes, to follow a corresponding trajectory within the simulated environment as conditioned on a respective body shape. The simulated human characters can have diverse characteristics (such as gender, body proportions, body shape, and so on) as observed in real-life crowds. A machine learning model can determine an action for a human character in a simulated environment, based at least on a humanoid state, a body shape, and task-related features. The task-related features can include an environmental feature and a trajectory.

公开(公告)号：US20230070514A1

公开(公告)日：2023-03-09

申请号：US17584213

申请日：2022-01-25

Applicant: NVIDIA Corporation

Inventor： Ye Yuan ,  Umar Iqbal ,  Pavlo Molchanov ,  Jan Kautz

IPC: G06T19/20 ,  G06T7/20 ,  G06T7/00

Abstract: In order to determine accurate three-dimensional (3D) models for objects within a video, the objects are first identified and tracked within the video, and a pose and shape are estimated for these tracked objects. A translation and global orientation are removed from the tracked objects to determine local motion for the objects, and motion infilling is performed to fill in any missing portions for the object within the video. A global trajectory is then determined for the objects within the video, and the infilled motion and global trajectory are then used to determine infilled global motion for the object within the video. This enables the accurate depiction of each object as a 3D pose sequence for that model that accounts for occlusions and global factors within the video.

公开(公告)号：US20240135618A1

公开(公告)日：2024-04-25

申请号：US18322319

申请日：2023-05-23

Applicant: NVIDIA Corporation

Inventor： Haotian Zhang ,  Ye Yuan ,  Jason Peng ,  Viktor Makoviichuk ,  Sanja Fidler

IPC: G06T13/40 ,  G06T7/70 ,  G06T17/00 ,  G06V40/20

CPC classification number: G06T13/40 ,  G06T7/70 ,  G06T17/00 ,  G06V40/23

Abstract: In various examples, artificial intelligence (AI) agents can be generated to synthesize more natural motion by simulated actors in various visualizations (such as video games or simulations). AI agents may employ one or more machine learning models and techniques, such as reinforcement learning, to enable synthesis of motion with enhanced realism. The AI agent can be trained based on widely-available broadcast video data, without the need for more costly and limited motion capture data. To account for the lower quality of such video data, various techniques can be employed, such as taking into account the motion of joints, and applying physics-based constraints on the actors, resulting in higher quality, more lifelike motion.

公开(公告)号：US20240070874A1

公开(公告)日：2024-02-29

申请号：US18135654

申请日：2023-04-17

Applicant: NVIDIA Corporation

Inventor： Muhammed Kocabas ,  Ye Yuan ,  Umar Iqbal ,  Pavlo Molchanov ,  Jan Kautz

IPC: G06T7/20 ,  G06T7/70

CPC classification number: G06T7/20 ,  G06T7/70 ,  G06T2207/20084 ,  G06T2207/30196 ,  G06T2207/30252 ,  G06T2210/12

Abstract: Estimating motion of a human or other object in video is a common computer task with applications in robotics, sports, mixed reality, etc. However, motion estimation becomes difficult when the camera capturing the video is moving, because the observed object and camera motions are entangled. The present disclosure provides for joint estimation of the motion of a camera and the motion of articulated objects captured in video by the camera.