公开(公告)号：US12260017B2

公开(公告)日：2025-03-25

申请号：US18182086

申请日：2023-03-10

Applicant: NVIDIA Corporation

Inventor： Nishant Puri

IPC: G06T7/11 ,  B60W30/08 ,  G06F3/01 ,  G06N3/08 ,  G06N20/00 ,  G06T7/00 ,  G06T7/20 ,  G06T7/70 ,  G06T7/73 ,  G06V10/764 ,  G06V10/82 ,  G06V20/56 ,  G06V20/59 ,  G06V40/16 ,  G06V40/18

Abstract: Machine learning systems and methods that determine gaze direction by using face orientation information, such as facial landmarks, to modify eye direction information determined from images of the subject's eyes. System inputs include eye crops of the eyes of the subject, as well as face orientation information such as facial landmarks of the subject's face in the input image. Facial orientation information, or facial landmark information, is used to determine a coarse prediction of gaze direction as well as to learn a context vector of features describing subject face pose. The context vector is then used to adaptively re-weight the eye direction features determined from the eye crops. The re-weighted features are then combined with the coarse gaze prediction to determine gaze direction.

公开(公告)号：US11934955B2

公开(公告)日：2024-03-19

申请号：US18051296

申请日：2022-10-31

Applicant: NVIDIA Corporation

Inventor： Nuri Murat Arar ,  Niranjan Avadhanam ,  Nishant Puri ,  Shagan Sah ,  Rajath Shetty ,  Sujay Yadawadkar ,  Pavlo Molchanov

IPC: G06N3/08 ,  G06F18/21 ,  G06F18/214 ,  G06N20/00 ,  G06V10/764 ,  G06V10/774 ,  G06V10/82 ,  G06V10/94 ,  G06V20/59 ,  G06V20/64 ,  G06V40/16 ,  G06V40/18

CPC classification number: G06N3/08 ,  G06F18/214 ,  G06F18/2193 ,  G06N20/00 ,  G06V10/764 ,  G06V10/774 ,  G06V10/82 ,  G06V10/95 ,  G06V20/597 ,  G06V20/647 ,  G06V40/171 ,  G06V40/193

Abstract: Systems and methods for more accurate and robust determination of subject characteristics from an image of the subject. One or more machine learning models receive as input an image of a subject, and output both facial landmarks and associated confidence values. Confidence values represent the degrees to which portions of the subject's face corresponding to those landmarks are occluded, i.e., the amount of uncertainty in the position of each landmark location. These landmark points and their associated confidence values, and/or associated information, may then be input to another set of one or more machine learning models which may output any facial analysis quantity or quantities, such as the subject's gaze direction, head pose, drowsiness state, cognitive load, or distraction state.

公开(公告)号：US11688074B2

公开(公告)日：2023-06-27

申请号：US17039437

申请日：2020-09-30

Applicant: NVIDIA Corporation

Inventor： Nishant Puri ,  Sakthivel Sivaraman ,  Rajath Shetty ,  Niranjan Avadhanam

IPC: G06T7/194 ,  G06T5/30 ,  G06T5/00 ,  G06N3/08 ,  G06V40/10 ,  G06F18/214 ,  G06F18/24

CPC classification number: G06T7/194 ,  G06F18/214 ,  G06F18/24 ,  G06N3/08 ,  G06T5/002 ,  G06T5/30 ,  G06V40/11 ,  G06V40/113 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/20132 ,  G06T2207/20221 ,  G06T2207/30196

Abstract: In various examples, a background of an object may be modified to generate a training image. A segmentation mask may be generated and used to generate an object image that includes image data representing the object. The object image may be integrated into a different background and used for data augmentation in training a neural network. Data augmentation may also be performed using hue adjustment (e.g., of the object image) and/or rendering three-dimensional capture data that corresponds to the object from selected views. Inference scores may be analyzed to select a background for an image to be included in a training dataset. Backgrounds may be selected and training images may be added to a training dataset iteratively during training (e.g., between epochs). Additionally, early or late fusion nay be employed that uses object mask data to improve inferencing performed by a neural network trained using object mask data.

公开(公告)号：US11841987B2

公开(公告)日：2023-12-12

申请号：US17751548

申请日：2022-05-23

Applicant: NVIDIA Corporation

Inventor： Hairong Jiang ,  Nishant Puri ,  Niranjan Avadhanam ,  Nuri Murat Arar

IPC: G06F3/01 ,  G06V40/19 ,  G06V40/18 ,  G06V10/44 ,  G06V10/764 ,  G06V10/82 ,  G06V40/16

CPC classification number: G06F3/013 ,  G06V10/454 ,  G06V10/764 ,  G06V10/82 ,  G06V40/171 ,  G06V40/18 ,  G06V40/19

Abstract: Machine learning systems and methods that learn glare, and thus determine gaze direction in a manner more resilient to the effects of glare on input images. The machine learning systems have an isolated representation of glare, e.g., information on the locations of glare points in an image, as an explicit input, in addition to the image itself. In this manner, the machine learning systems explicitly consider glare while making a determination of gaze direction, thus producing more accurate results for images containing glare.

公开(公告)号：US20230078171A1

公开(公告)日：2023-03-16

申请号：US18051296

申请日：2022-10-31

Applicant: NVIDIA Corporation

Inventor： Nuri Murat Arar ,  Niranjan Avadhanam ,  Nishant Puri ,  Shagan Sah ,  Rajath Shetty ,  Sujay Yadawadkar ,  Pavlo Molchanov

IPC: G06K9/62 ,  G06N20/00 ,  G06V10/94 ,  G06V20/59 ,  G06V20/64 ,  G06V40/16 ,  G06V40/18

Abstract: Systems and methods for more accurate and robust determination of subject characteristics from an image of the subject. One or more machine learning models receive as input an image of a subject, and output both facial landmarks and associated confidence values. Confidence values represent the degrees to which portions of the subject's face corresponding to those landmarks are occluded, i.e., the amount of uncertainty in the position of each landmark location. These landmark points and their associated confidence values, and/or associated information, may then be input to another set of one or more machine learning models which may output any facial analysis quantity or quantities, such as the subject's gaze direction, head pose, drowsiness state, cognitive load, or distraction state.

公开(公告)号：US20220300072A1

公开(公告)日：2022-09-22

申请号：US17206585

申请日：2021-03-19

Applicant: NVIDIA Corporation

Inventor： Nuri Murat Arar ,  Sujay Yadawadkar ,  Hairong Jiang ,  Nishant Puri ,  Niranjan Avadhanam

IPC: G06F3/01 ,  G06K9/46 ,  G06K9/62 ,  G06K9/00

Abstract: In various examples, systems and methods are disclosed that provide highly accurate gaze predictions that are specific to a particular user by generating and applying, in deployment, personalized calibration functions to outputs and/or layers of a machine learning model. The calibration functions corresponding to a specific user may operate on outputs (e.g., gaze predictions from a machine learning model) to provide updated values and gaze predictions. The calibration functions may also be applied one or more last layers of the machine learning model to operate on features identified by the model and provide values that are more accurate. The calibration functions may be generated using explicit calibration methods by instructing users to gaze at a number of identified ground truth locations within the interior of the vehicle. Once generated, the calibration functions may be modified or refined through implicit gaze calibration points and/or regions based on gaze saliency maps.

公开(公告)号：US20240412491A1

公开(公告)日：2024-12-12

申请号：US18207953

申请日：2023-06-09

Applicant: NVIDIA Corporation

Inventor： Shagan Sah ,  Nishant Puri ,  Yuzhuo Ren ,  Rajath Bellipady Shetty ,  Weili Nie ,  Arash Vahdat ,  Animashree Anandkumar

IPC: G06V10/776 ,  G06N3/094 ,  G06T11/00 ,  G06V10/75 ,  G06V10/774 ,  G06V10/82 ,  G06V40/16

Abstract: Apparatuses, system, and techniques use one or more first neural networks to generate one or more synthetic data to train one or more second neural networks based, at least in part, on one or more performance metrics of one or more second neural networks.

公开(公告)号：US20230244941A1

公开(公告)日：2023-08-03

申请号：US18298115

申请日：2023-04-10

Applicant: NVIDIA Corporation

Inventor： Nuri Murat Arar ,  Hairong Jiang ,  Nishant Puri ,  Rajath Shetty ,  Niranjan Avadhanam

IPC: G06N3/08 ,  G06N20/00 ,  G06V10/94 ,  G06V20/59 ,  G06V20/64 ,  G06V40/16 ,  G06V40/18 ,  G06F18/214 ,  G06F18/21 ,  G06V10/764 ,  G06V10/774 ,  G06V10/82

CPC classification number: G06N3/08 ,  G06F18/214 ,  G06F18/2193 ,  G06N20/00 ,  G06V10/82 ,  G06V10/95 ,  G06V10/764 ,  G06V10/774 ,  G06V20/597 ,  G06V20/647 ,  G06V40/171 ,  G06V40/193

Abstract: Systems and methods for determining the gaze direction of a subject and projecting this gaze direction onto specific regions of an arbitrary three-dimensional geometry. In an exemplary embodiment, gaze direction may be determined by a regression-based machine learning model. The determined gaze direction is then projected onto a three-dimensional map or set of surfaces that may represent any desired object or system. Maps may represent any three-dimensional layout or geometry, whether actual or virtual. Gaze vectors can thus be used to determine the object of gaze within any environment. Systems can also readily and efficiently adapt for use in different environments by retrieving a different set of surfaces or regions for each environment.

公开(公告)号：US11340701B2

公开(公告)日：2022-05-24

申请号：US16902737

申请日：2020-06-16

Applicant: NVIDIA Corporation

Inventor： Hairong Jiang ,  Nishant Puri ,  Niranjan Avadhanam ,  Nuri Murat Arar

IPC: G06F3/01 ,  G06K9/00

Abstract: Machine learning systems and methods that learn glare, and thus determine gaze direction in a manner more resilient to the effects of glare on input images. The machine learning systems have an isolated representation of glare, e.g., information on the locations of glare points in an image, as an explicit input, in addition to the image itself. In this manner, the machine learning systems explicitly consider glare while making a determination of gaze direction, thus producing more accurate results for images containing glare.

公开(公告)号：US20220101047A1

公开(公告)日：2022-03-31

申请号：US17039437

申请日：2020-09-30

Applicant: NVIDIA Corporation

Inventor： Nishant Puri ,  Sakthivel Sivaraman ,  Rajath Shetty ,  Niranjan Avadhanam

IPC: G06K9/62 ,  G06T7/194 ,  G06K9/00 ,  G06T5/30 ,  G06T5/00 ,  G06N3/08

Abstract: In various examples, a background of an object may be modified to generate a training image. A segmentation mask may be generated and used to generate an object image that includes image data representing the object. The object image may be integrated into a different background and used for data augmentation in training a neural network. Data augmentation may also be performed using hue adjustment (e.g., of the object image) and/or rendering three-dimensional capture data that corresponds to the object from selected views. Inference scores may be analyzed to select a background for an image to be included in a training dataset. Backgrounds may be selected and training images may be added to a training dataset iteratively during training (e.g., between epochs). Additionally, early or late fusion nay be employed that uses object mask data to improve inferencing performed by a neural network trained using object mask data.