公开(公告)号：US20230132041A1

公开(公告)日：2023-04-27

申请号：US18047494

申请日：2022-10-18

Applicant: GOOGLE LLC

Inventor： Alex Olwal ,  Ruofei Du

IPC: G06F3/01 ,  G02B27/01 ,  G06F3/16

Abstract: The disclosed systems and method correlates user behaviors with audio processing to achieve more accurate conclusions about sounds in a user's environment. These conclusions may, in turn, be used to adjust the way a device, such as AR glasses, operate or respond to the sounds. For example, audio events determined from processing speech can be correlated with behavior events determined by sensing a user to improve a speech-to-text transcript of the speech by separating, or otherwise altering, the text in the transcript by speaker.

公开(公告)号：US20230071778A1

公开(公告)日：2023-03-09

申请号：US17446877

申请日：2021-09-03

Applicant: Google LLC

Inventor： Ruofei Du ,  Hendrik Wagenaar ,  Alex Olwal

IPC: G10L21/10 ,  G06K9/00 ,  G06F3/01 ,  H04R3/00 ,  H04R1/40 ,  G06T7/70 ,  G10L15/26 ,  G10L15/22 ,  H04R1/08 ,  G10L15/25 ,  G10L21/18 ,  G06T7/50

Abstract: An augmented reality (AR) device, such as AR glasses, may include a microphone array. The sensitivity of the microphone array can be directed to a target by beamforming, which includes combining the audio of each microphone of the array in a particular way based on a location of the target. The present disclosure describes systems and methods to determine the location of the target based on a gaze of a user and beamform the audio accordingly. This eye-tracked beamforming (i.e., foveated beamforming) can be used by AR applications to enhance sounds from a gaze direction and to suppress sounds from other directions. Additionally, the gaze information can be used to help visualize the results of an AR application, such as speech-to-text.

公开(公告)号：US20250140264A1

公开(公告)日：2025-05-01

申请号：US18996721

申请日：2022-07-21

Applicant: GOOGLE LLC

Inventor： Isaac Allen Fehr ,  Angela Krone ,  Dongeek Shin ,  Ruofei Du

IPC: G10L17/26 ,  G06F3/01 ,  G10L15/22

Abstract: A system for controlling a head mounted device (HMD). The system includes a processor of a controller of the HMD connected to an Internal Measurement Unit (IMU) and a memory on which are stored machine-readable instructions that when executed by the processor, cause the processor to: receive Internal Measurement Unit (IMU) data generated in response to vibrations produced by voiced nasal consonant vocalizations produced by an HMD user, analyze the IMU data to determine whether the IMU data corresponds to an HMD control command, and responsive to a determination that the IMU data corresponds to the HMD control command, execute the HMD control command.

公开(公告)号：US20250094116A1

公开(公告)日：2025-03-20

申请号：US18884920

申请日：2024-09-13

Applicant: Google LLC

Inventor： Ruofei Du ,  David Kim ,  Fengyuan Zhu ,  Daniel Kalmar

IPC: G06F3/14 ,  G06F3/01 ,  G06F3/0346 ,  G06F3/04886

Abstract: A method can include determining, by a head-mounted device mounted on a head of a user, a number of degrees of freedom of an input modality of an auxiliary device; determining an input modality of the head-mounted device based on the number of degrees of freedom of the input modality of the auxiliary device; and presenting the input modality of the head-mounted device to the user.

公开(公告)号：US12236541B2

公开(公告)日：2025-02-25

申请号：US17998736

申请日：2020-05-29

Applicant: GOOGLE LLC

Inventor： David Kim ,  Ruofei Du

IPC: G06T19/00 ,  G06T7/50 ,  G06T15/06 ,  G06T15/60 ,  G06T17/20

Abstract: Techniques of introducing virtual objects into a physical environment of AR system include displacing vertices of a mesh representing the physical environment based on a live depth map. For example, an AR system generates a mesh template, i.e., an initial mesh with vertices that represents a physical environment and a depth map that indicates a geometry of real objects within the physical environment. The AR system is configured to represent the real objects in the physical environment by displacing the vertices of the mesh based on depth values of the depth map and parameter values of a pinhole camera model. The depth values may be taken from the perspective of an illumination source in the physical environment.

公开(公告)号：US20240303918A1

公开(公告)日：2024-09-12

申请号：US18484783

申请日：2023-10-11

Applicant: GOOGLE LLC

Inventor： Ruofei Du ,  Xun Qian ,  Yinda Zhang ,  Alex Olwal

IPC: G06T17/00 ,  G06T7/55 ,  G06T7/73

CPC classification number: G06T17/00 ,  G06T7/55 ,  G06T7/73 ,  G06T2207/10016 ,  G06T2207/10024 ,  G06T2207/20081 ,  G06T2207/30201 ,  G06T2210/22 ,  H04N23/611

Abstract: A method can include receiving, via a camera, a first video stream of a face of a user; determining a location of the face of the user based on the first video stream and a facial landmark detection model; receiving, via the camera, a second video stream of the face of the user; generating a depth map based on the second video stream, the location of the face of the user, and a depth prediction model; and generating a representation of the user based on the depth map and the second video stream.

公开(公告)号：US20240303908A1

公开(公告)日：2024-09-12

申请号：US18547628

申请日：2021-04-30

Applicant: GOOGLE LLC

Inventor： Yinda Zhang ,  Danhang Tang ,  Ruofei Du ,  Zhang Chen ,  Kyle Genova ,  Sofien Bouaziz ,  Thomas Allen Funkhouser ,  Sean Ryan Francesco Fanello ,  Christian Haene

IPC: G06T15/08 ,  G06T3/40 ,  G06T9/00 ,  G06T19/20

CPC classification number: G06T15/08 ,  G06T3/40 ,  G06T9/00 ,  G06T19/20

Abstract: A method including generating a first vector based on a first grid and a three-dimensional (3D) position associated with a first implicit representation (IR) of a 3D object, generating at least one second vector based on at least one second grid and an upsampled first grid, decoding the first vector to generate a second IR of the 3D object, decoding the at least one second vector to generate at least one third IR of the 3D object, generating a composite IR of the 3D object based on the second IR of the 3D object and the at least one third IR of the 3D object, and generating a reconstructed volume representing the 3D object based on the composite IR of the 3D object.

公开(公告)号：US20240265605A1

公开(公告)日：2024-08-08

申请号：US18165779

申请日：2023-02-07

Applicant: GOOGLE LLC

Inventor： Ruofei Du ,  Xingyu Liu

IPC: G06T13/20 ,  G06T13/40 ,  G06T13/80 ,  G10L25/63

CPC classification number: G06T13/205 ,  G06T13/40 ,  G06T13/80 ,  G10L25/63

Abstract: A system and method may receive audio signal information associated with a user. An expression prediction may be determined by executing an expression determination model using the audio signal information as input. An avatar animation may be generated based on the expression prediction, where the avatar animation includes non-verbal expression representing the expression prediction.

公开(公告)号：US11994675B2

公开(公告)日：2024-05-28

申请号：US17663444

申请日：2022-05-16

Applicant: GOOGLE LLC

Inventor： Ruofei Du ,  Maksym Dzitsiuk

IPC: G02B27/00 ,  G08B25/01 ,  G02B27/01

CPC classification number: G02B27/0093 ,  G08B25/016 ,  G02B2027/014 ,  G02B2027/0178

Abstract: A head-mounted device, such as augmented reality glasses, can be configured to track screen time. The head-mounted device can advantageously track screen time across a variety of devices whether or not they are configured to such measurements. Various modes for tracking the screen time may be used based on accuracy, power consumption, and privacy. The head-mounted device may track in a mode based on a battery level, a device setting or a user input. The head-mounted device may accumulate screen times in a database and in some implementations, can link screen times with devices in the database based on device information gathered by the head-mounted device during the tracking process. The head-mounted device may be configured to generate and display an alert based on one or more rules about screen time.

公开(公告)号：US11954899B2

公开(公告)日：2024-04-09

申请号：US18274371

申请日：2021-03-11

Applicant: Google LLC

Inventor： Yinda Zhang ,  Feitong Tan ,  Danhang Tang ,  Mingsong Dou ,  Kaiwen Guo ,  Sean Ryan Francesco Fanello ,  Sofien Bouaziz ,  Cem Keskin ,  Ruofei Du ,  Rohit Kumar Pandey ,  Deqing Sun

IPC: G06V10/771 ,  G06T7/70 ,  G06T17/00 ,  G06V10/44 ,  G06V10/75

CPC classification number: G06V10/771 ,  G06T7/70 ,  G06T17/00 ,  G06V10/44 ,  G06V10/751 ,  G06T2207/20081 ,  G06T2207/20084

Abstract: Systems and methods for training models to predict dense correspondences across images such as human images. A model may be trained using synthetic training data created from one or more 3D computer models of a subject. In addition, one or more geodesic distances derived from the surfaces of one or more of the 3D models may be used to generate one or more loss values, which may in turn be used in modifying the model's parameters during training.