公开(公告)号：US20220222869A1

公开(公告)日：2022-07-14

申请号：US17707774

申请日：2022-03-29

Applicant: Apple Inc.

Inventor： Ian M. Richter ,  Daniel Ulbricht ,  Jean-Daniel E. Nahmias ,  Omar Elafifi ,  Peter Meier

IPC: G06T11/00

Abstract: In one implementation, a method includes: obtaining a user input to view SR content associated with video content; if the video content includes a first scene when the user input was detected: obtaining first SR content for a first time period of the video content associated with the first scene; obtaining a task associated with the first scene; and causing presentation of the first SR content and a first indication of the task associated with the first scene; and if the video content includes a second scene when the user input was detected: obtaining second SR content for a second time period of the video content associated with the second scene; obtaining a task associated with the second scene; and causing presentation of the second SR content and a second indication of the task associated with the second scene.

公开(公告)号：US11373377B2

公开(公告)日：2022-06-28

申请号：US17203393

申请日：2021-03-16

Applicant: Apple Inc.

Inventor： Bo Morgan ,  Mark E. Drummond ,  Peter Meier ,  Cameron J. Dunn ,  John Christopher Russell ,  Siva Chandra Mouli Sivapurapu ,  Ian M. Richter

IPC: G06T19/00 ,  A63F13/55 ,  G06F40/14 ,  G06F3/01

Abstract: In various implementations, a device surveys a scene and presents, within the scene, a extended reality (XR) environment including one or more assets that evolve over time (e.g., change location or age). Modeling such an XR environment at various timescales can be computationally intensive, particularly when modeling the XR environment over larger timescales. Accordingly, in various implementations, different models are used to determine the environment state of the XR environment when presenting the XR environment at different timescales.

公开(公告)号：US11328456B2

公开(公告)日：2022-05-10

申请号：US16963308

申请日：2019-01-18

Applicant: Apple Inc.

Inventor： Ian M. Richter ,  Daniel Ulbricht ,  Jean-Daniel E. Nahmias ,  Omar Elafifi ,  Peter Meier

IPC: G06T11/00 ,  G06F3/04883

Abstract: In one implementation, a method includes: while causing presentation of video content having a current plot setting, receiving a user input indicating a request to explore the current plot setting; obtaining synthesized reality (SR) content associated with the current plot setting in response to receiving the user input; causing presentation of the SR content associated with the current plot setting; receiving one or more user interactions with the SR content; and adjusting the presentation of the SR content in response to receiving the one or more user interactions with the SR content.

公开(公告)号：US11080932B2

公开(公告)日：2021-08-03

申请号：US16524787

申请日：2019-07-29

Applicant: Apple Inc.

Inventor： Peter Meier ,  Stefan Holzer

IPC: G06T19/00 ,  G06T15/20 ,  G06T7/73 ,  G06T15/04

Abstract: The invention relates to a method for representing a virtual object in a real environment, having the following steps: generating a two-dimensional image of a real environment by means of a recording device, ascertaining a position of the recording device relative to at least one component of the real environment, segmenting at least one area of the real environment unmarked in reality in the two-dimensional image for identifying at least one segment of the real environment in distinction to a remaining part of the real environment while supplying corresponding segmentation data, and merging the virtual object with the two-dimensional image of the real environment with consideration of the segmentation data such that at least one part of the segment of the real environment is removed from the image of the real environment. The invention permits any collisions of virtual objects with real objects that occur upon merging with a real environment to be represented in a way largely close to reality.

公开(公告)号：US20210183136A1

公开(公告)日：2021-06-17

申请号：US17129340

申请日：2020-12-21

Applicant: Apple Inc.

Inventor： Lejing Wang ,  Peter Meier ,  Stefan Misslinger

IPC: G06T15/20 ,  G06T15/06 ,  G06T19/00

Abstract: A method for representing virtual information in a view of a real environment is provided that includes: providing a system setup including at least one display device, wherein the system setup is adapted for blending in virtual information on the display device in at least part of the view, determining a position and orientation of a viewing point relative to at least one component of the real environment, providing a geometry model of the real environment, providing at least one item of virtual information and a position of the at least one item of virtual information, determining whether the position of the item of virtual information is inside a 2D or 3D geometrical shape, determining a criterion which is indicative of whether the built-in real object is at least partially visible or non-visible in the view of the real environment, and blending in the at least one item of virtual information on the display device in at least part of the view of the real environment.

公开(公告)号：US20210064910A1

公开(公告)日：2021-03-04

申请号：US17001788

申请日：2020-08-25

Applicant: Apple Inc.

Inventor： Peter Meier ,  Daniel Kurz ,  Brian Chris Clark ,  Mohamed Selim Ben Himane

IPC: G06K9/46 ,  G06K9/20 ,  G06T7/73 ,  G06K9/00

Abstract: Various implementations disclosed herein include devices, systems, and methods that detect surfaces and reflections in such surfaces. Some implementations involve providing a CGR environment that includes virtual content that replaces the appearance of a user or the user's device in a mirror or other surface providing a reflection. For example, a CGR environment may be modified to include a reflection of the user that does not include the device that the user is holding or wearing. In another example, the CGR environment is modified so that virtual content, such as a newer version of the electronic device or a virtual wand, replaces the electronic device in the reflection. In another example, the CGR environment is modified so that virtual content, such as a user avatar, replaces the user in the reflection.

公开(公告)号：US10819962B2

公开(公告)日：2020-10-27

申请号：US14758121

申请日：2012-12-28

Applicant: Apple Inc.

Inventor： Peter Meier ,  Mohamed Selim Ben Himane ,  Daniel Kurz

IPC: H04N9/31 ,  H04N5/225 ,  H04N5/33 ,  H04N13/128 ,  H04N13/15 ,  H04N13/257 ,  G01B11/24 ,  G01B11/25 ,  H04N13/00

Abstract: A method of projecting digital information on a real object in a real environment includes the steps of projecting digital information on a real object or part of a real object with a visible light projector, capturing at least one image of the real object with the projected digital information using a camera, providing a depth sensor registered with the camera, the depth sensor capturing depth data of the real object or part of the real object, and calculating a spatial transformation between the visible light projector and the real object based on the at least one image and the depth data. The invention is also concerned with a corresponding system.

公开(公告)号：US20200242846A1

公开(公告)日：2020-07-30

申请号：US16843462

申请日：2020-04-08

Applicant: Apple Inc.

Inventor： Peter Meier ,  Stefan Misslinger ,  Anton Fedosov

IPC: G06T19/00 ,  G06T15/20 ,  G06F3/01

Abstract: The invention relates to a method for ergonomically representing virtual information in a real environment, including the following steps: providing at least one view of a real environment and of a system setup for blending in virtual information for superimposing with the real environment in at least part of the view, the system setup comprising at least one display device, ascertaining a position and orientation of at least one part of the system setup relative to at least one component of the real environment, subdividing at least part of the view of the real environment into a plurality of regions comprising a first region and a second region, with objects of the real environment within the first region being placed closer to the system setup than objects of the real environment within the second region, and blending in at least one item of virtual information on the display device in at least part of the view of the real environment, considering the position and orientation of said at least one part of the system setup, wherein the virtual information is shown differently in the first region than in the second region with respect to the type of blending in the view of the real environment.

公开(公告)号：US20200074280A1

公开(公告)日：2020-03-05

申请号：US16513991

申请日：2019-07-17

Applicant: Apple Inc.

Inventor： Peter Meier ,  Tanmay Batra

IPC: G06N3/04 ,  G06N3/08 ,  G06K9/62 ,  G06T7/70 ,  G06T7/20

Abstract: In some implementations a neural network is trained to perform a main task using a clustering constraint, for example, using both a main task training loss and a clustering training loss. Training inputs are inputted into a main task neural network to produce output labels predicting locations of the parts of the objects in the training inputs. Data from pooled layers of the main task neural network is inputted into a clustering neural network. The main task neural network and the clustering neural network are trained based on a main task loss from the main task neural network and a clustering loss from the clustering neural network. The main task loss is determined by comparing differences between the output labels and the training labels. The clustering loss encourages the clustering network to learn to label the parts of the objects individually, e.g., to learn groups corresponding to the object parts.

公开(公告)号：US20200065653A1

公开(公告)日：2020-02-27

申请号：US16515397

申请日：2019-07-18

Applicant: Apple Inc.

Inventor： Peter Meier ,  Tanmay Batra

IPC: G06N3/04 ,  G06N3/08 ,  G06K9/62 ,  G06T7/70 ,  G06T7/20 ,  G06T7/10

Abstract: In some implementations at an electronic device, training a dual EDNN includes defining a data structure of attributes corresponding to defined parts of a task, processing a first instance of an input using a first EDNN to produce a first output while encoding a first set of the attributes in a first latent space, and processing a second instance of the input using a second EDNN to produce a second output while encoding attribute differences from attribute averages in a second latent space. The device then determines a second set of the attributes based on the attribute differences and the attribute averages. The device then adjusts parameters of the first and second EDNNs based on comparing the first instance of the input to the first output, the second instance of the input to the second output, and the first set of attributes to the second set of attributes.