公开(公告)号：US11928780B2

公开(公告)日：2024-03-12

申请号：US17869183

申请日：2022-07-20

Applicant: Apple Inc.

Inventor： Payal Jotwani

IPC: G06T17/00 ,  G06F18/21 ,  G06F18/232 ,  G06F18/24 ,  G06T7/50 ,  G06T15/20 ,  G06T17/20 ,  G06T19/20 ,  G06V20/00

CPC classification number: G06T17/20 ,  G06F18/2163 ,  G06F18/232 ,  G06F18/24 ,  G06T7/50 ,  G06T15/20 ,  G06T19/20 ,  G06V20/00 ,  G06T2207/10028 ,  G06T2210/56

Abstract: In one implementation, a method of enriching a three-dimensional scene model with a three-dimensional object model based on a semantic label is performed at a device including one or more processors and non-transitory memory. The method includes obtaining a three-dimensional scene model of a physical environment including a plurality of points, wherein each of the plurality of points is associated with a set of coordinates in a three-dimensional space, wherein a subset of the plurality of points is associated with a particular cluster identifier and a particular semantic label. The method includes retrieving a three-dimensional object model based on the particular semantic label, the three-dimensional object model including at least a plurality of points. The method includes updating the three-dimensional scene model by replacing the subset of the plurality of points with the three-dimensional object model.

公开(公告)号：US11804012B1

公开(公告)日：2023-10-31

申请号：US17323799

申请日：2021-05-18

Applicant: Apple Inc.

Inventor： Daniel Laszlo Kovacs ,  Payal Jotwani ,  Dan Feng

IPC: G06T19/00 ,  G06T17/20

CPC classification number: G06T19/003 ,  G06T17/20

Abstract: In some implementations, a method of navigation mesh exploration is performed at a virtual agent operating system. The method includes: determining one or more first sensory perception regions for one or more senses of a virtual agent based on a first perceptual vector associated with the virtual agent; generating a first portion of a navigation mesh for the XR environment based on the one or more first sensory perception regions, wherein the first portion of the navigation mesh includes candidate subsequent locations different from the first location; and in response to detecting movement of the virtual agent to a respective candidate subsequent location among candidate subsequent locations, generating a second portion of the navigation mesh for the XR environment based on one or more second sensory perception regions for the one or more senses of the virtual agent relative to the respective candidate subsequent location.

公开(公告)号：US20210097731A1

公开(公告)日：2021-04-01

申请号：US16987805

申请日：2020-08-07

Applicant: Apple Inc.

Inventor： Payal Jotwani

IPC: G06T11/00 ,  G06K9/00 ,  G06T7/60 ,  G06F16/53 ,  G06F16/58

Abstract: Various implementations disclosed herein include devices, systems, and methods for generating a dimensionally accurate computer-generated reality (CGR) environment with a scaled CGR object. In some implementations, a method includes obtaining environmental data corresponding to a physical environment. A known physical article located within the physical environment is identified based on the environmental data. The known physical article is associated with a known dimension. A physical dimension of the physical environment is determined based on the known dimension of the known physical article. A CGR environment is generated that represents the physical environment. A virtual dimension of the CGR environment is a function of the physical dimension of the physical environment.

公开(公告)号：US11948380B1

公开(公告)日：2024-04-02

申请号：US17158212

申请日：2021-01-26

Applicant: Apple Inc.

Inventor： Payal Jotwani

IPC: G06V30/262 ,  G06F18/24 ,  G06T7/10 ,  G06T7/70 ,  G06T7/80

CPC classification number: G06V30/274 ,  G06F18/24 ,  G06T7/10 ,  G06T7/70 ,  G06T7/80 ,  G06T2207/10028

Abstract: In one implementation, a method of determining an estimate of a position of a camera is performed at a device including one or more processors and non-transitory memory. The method includes obtaining a point cloud of a physical environment including a plurality of points, wherein each of the plurality of points is associated with set of three-dimensional coordinates in a three-dimensional coordinate system of the physical environment, wherein the plurality of points includes a first cluster of points associated with a first semantic label. The method includes obtaining a two-dimensional image of the physical environment associated with a point-of-view set of three-dimensional coordinates in the three-dimensional coordinate system of the physical environment. The method includes detecting, in the two-dimensional image of the physical environment, a representation of a first object corresponding to the first semantic label. The method includes determining a first plurality of sets of two-dimensional coordinates in a two-dimensional coordinate system of the two-dimensional image of the physical environment corresponding to the representation of the first object. The method includes determining, from the first cluster of points associated with the first semantic label, a first plurality of sets of three-dimensional coordinates in the three-dimensional coordinate system of the physical environment respectively corresponding to the first plurality of sets of two-dimensional coordinates in the two-dimensional coordinate system of the two-dimensional image of the physical environment. The method includes determining an estimate of the point-of-view set of three-dimensional coordinates based on the first plurality of two-dimensional coordinates and the first plurality of three-dimensional coordinates.

公开(公告)号：US11868526B2

公开(公告)日：2024-01-09

申请号：US17703278

申请日：2022-03-24

Applicant: Apple Inc.

Inventor： Michael J. Gutensohn ,  Payal Jotwani ,  Mark E. Drummond ,  Daniel L. Kovacs

IPC: G06F3/01

CPC classification number: G06F3/013

Abstract: In one implementation, a method for recording an XR environment. The method includes: presenting, via the display device, a graphical environment with one or more virtual agents, wherein the graphical environment corresponds to a composition of extended reality (XR) content, including the one or more virtual agents, and an image stream of a physical environment captured from a first point-of-view (POV) of the physical environment; detecting, via the one or more input devices, a user input selecting a first virtual agent from among the one or more virtual agents; and in response to detecting the user input, recording a plurality of data streams associated with the graphical environment including a first image stream of the graphical environment from the first POV and one or more data streams of the graphical environment from a current POV of the first virtual agent.

公开(公告)号：US20230350536A1

公开(公告)日：2023-11-02

申请号：US18113020

申请日：2023-02-22

Applicant: Apple Inc.

Inventor： Dan Feng ,  Aashi Manglik ,  Adam M. O'Hern ,  Bo Morgan ,  Bradley W. Peebler ,  Daniel L. Kovacs ,  Edward Ahn ,  James Moll ,  Mark E. Drummond ,  Michelle Chua ,  Mu Qiao ,  Noah Gamboa ,  Payal Jotwani ,  Siva Chandra Mouli Sivapurapu

IPC: G06F3/04815 ,  G06T17/00 ,  G10L15/22 ,  G06F3/01

CPC classification number: G06F3/04815 ,  G06T17/00 ,  G10L15/22 ,  G06F3/017 ,  G10L2015/223

Abstract: Various implementations disclosed herein include devices, systems, and methods for selecting a point-of-view (POV) for displaying an environment. In some implementations, a device includes a display, one or more processors, and a non-transitory memory. In some implementations, a method includes obtaining a request to display a graphical environment. The graphical environment is associated with a set of saliency values corresponding to respective portions of the graphical environment. A POV for displaying the graphical environment is selected based on the set of saliency values. The graphical environment is displayed from the selected POV on the display.

公开(公告)号：US11763525B1

公开(公告)日：2023-09-19

申请号：US17158219

申请日：2021-01-26

Applicant: Apple Inc.

Inventor： Payal Jotwani

IPC: G06T17/20 ,  G06V10/75 ,  G06F18/23 ,  G06T19/00

CPC classification number: G06T17/205 ,  G06F18/23 ,  G06V10/751 ,  G06T19/006

Abstract: In one implementation, a method of blind object tracking is performed at a device including one or more processors and non-transitory memory. The method includes obtaining a first three-dimensional scene model of a physical environment during a first time period including a first plurality of points, wherein each of the first plurality of points is associated with a set of coordinates in a three-dimensional space, wherein a subset of the first plurality of points is associated with a particular cluster identifier. The method includes obtaining a second three-dimensional scene model of the physical environment during a second time period subsequent to the first time period including a second plurality of points, wherein each of the second plurality of points is associated with a set of coordinates the three-dimensional space. The method includes associating a subset of the second plurality of points with the particular cluster identifier, wherein the subset of the second plurality of points includes points with different sets of coordinates in the three-dimensional space than the points of the subset of the first plurality of points.

公开(公告)号：US11710276B1

公开(公告)日：2023-07-25

申请号：US17358876

申请日：2021-06-25

Applicant: Apple Inc.

Inventor： Daniel Laszlo Kovacs ,  Siva Chandra Mouli Sivapurapu ,  Payal Jotwani ,  Noah Jonathan Gamboa

IPC: G06T17/00 ,  G06N3/00 ,  G06T19/00

CPC classification number: G06T19/003

Abstract: In one implementation, a method for improved motion planning. The method includes: obtaining a macro task for a virtual agent within a virtual environment; generating a search-tree based on at least one of the macro task, a state of the virtual environment, and a state of the virtual agent, wherein the search-tree includes a plurality of task nodes corresponding to potential tasks for performance by the virtual agent in furtherance of the macro task; and determining physical motion plans (PMPs) for at least some of the plurality of task nodes within the search-tree in order to generate a lookahead planning gradient for the first time, wherein a granularity of a PMP for a respective task node in the first search-tree is a function of the temporal distance of the respective task node from the first time.

公开(公告)号：US20250054249A1

公开(公告)日：2025-02-13

申请号：US18926113

申请日：2024-10-24

Applicant: Apple Inc.

Inventor： Mark E. Drummond ,  Daniel L. Kovacs ,  Shaun D. Budhram ,  Edward Ahn ,  Behrooz Mahasseni ,  Aashi Manglik ,  Payal Jotwani ,  Mu Qiao ,  Bo Morgan ,  Noah Gamboa ,  Michael J. Gutensohn ,  Dan Feng ,  Siva Chandra Mouli Sivapurapu

IPC: G06T19/00

Abstract: In one implementation, a method of displaying content is performed at a device including a display, one or more processors, and non-transitory memory. The method includes scanning a first physical environment to detect a first physical object in the first physical environment and a second physical object in the first physical environment, wherein the first physical object meets at least one first object criterion and the second physical object meets at least one second object criterion. The method includes displaying, in association with the first physical environment, a virtual object moving along a first path from the first physical object to the second physical object. The method includes scanning a second physical environment to detect a third physical object in the second physical environment and a fourth physical object in the second physical environment, wherein the third physical object meets the at least one first object criterion and the fourth physical object meets the at least one second object criterion. The method includes displaying, in association with the second physical environment, the virtual object moving along a second path from the third physical object to the fourth physical object, wherein the second path is different than the first path.

公开(公告)号：US12182327B2

公开(公告)日：2024-12-31

申请号：US18523324

申请日：2023-11-29

Applicant: Apple Inc.

Inventor： Michael J. Gutensohn ,  Payal Jotwani ,  Mark E. Drummond ,  Daniel L. Kovacs

IPC: G06F3/01

Abstract: In one implementation, a method for recording an XR environment. The method includes: presenting, via the display device, a graphical environment with one or more virtual agents, wherein the graphical environment corresponds to a composition of extended reality (XR) content, including the one or more virtual agents, and an image stream of a physical environment captured from a first point-of-view (POV) of the physical environment; detecting, via the one or more input devices, a user input selecting a first virtual agent from among the one or more virtual agents; and in response to detecting the user input, recording a plurality of data streams associated with the graphical environment including a first image stream of the graphical environment from the first POV and one or more data streams of the graphical environment from a current POV of the first virtual agent.