公开(公告)号：US10659750B2

公开(公告)日：2020-05-19

申请号：US15328349

申请日：2014-07-23

Applicant: Apple Inc.

Inventor： Peter Meier ,  Lejing Wang ,  Stefan Misslinger

IPC: H04N13/111 ,  H04N13/221 ,  G06T19/00 ,  G06T7/73 ,  H04N13/271 ,  H04N13/00

Abstract: The disclosure relates to a method and system for presenting at least part of an image of a real object in a view of a real environment, comprising providing a first image of at least part of a real object captured by a first camera, determining at least part of the real object in the first image as an object image area, determining a first 3D plane relative to the first camera, the first camera being at a position where the first image is captured, providing at least one image feature related to the real object in the first image, providing at least one first ray passing an optical center of the first camera being at a position where the first image is captured and the at least one image feature, determining, according to a first plane normal direction of the first 3D plane, at least one first angle between the first 3D plane and the at least one first ray, providing a second image of a real environment captured by a second camera, determining a second 3D plane relative to the second camera, the second camera being at a position where the second image is captured, mapping at least part of the object image area to a target space in a view of the real environment captured at a viewpoint, wherein the at least one image feature is mapped to at least one target position in a coordinate system of the view, determining at least one second angle between the second 3D plane and at least one second ray passing the viewpoint of capturing the view and the at least one target position according to a second plane normal direction of the second 3D plane, wherein the at least one image feature is mapped to the at least one target position such that a difference between the at least one first angle and the at least one second angle is minimized.

公开(公告)号：US09934612B2

公开(公告)日：2018-04-03

申请号：US14633386

申请日：2015-02-27

Applicant: Apple Inc.

Inventor： Peter Meier ,  Selim Ben Himane ,  Stefan Misslinger ,  Ben Blachnitzky

IPC: H04N7/18 ,  G06T19/00 ,  H04N5/222 ,  G06T15/00 ,  G06T7/73

CPC classification number: G06T19/006 ,  G06T7/75 ,  G06T15/00 ,  G06T2207/20092 ,  G06T2207/30244 ,  G06T2210/04 ,  H04N5/2224

Abstract: Method for determining the pose of a camera with respect to an object of a real environment for use in authoring/augmented reality application that includes generating a first image by the camera capturing a real object of a real environment, generating first orientation data from at least one orientation sensor associated with the camera or from an algorithm which analyzes the first image for finding and determining features which are indicative of an orientation of the camera, allocating a distance of the camera to the real object, generating distance data indicative of the allocated distance, determining the pose of the camera with respect to a coordinate system related to the real object of the real environment using the distance data and the first orientation data. May be performed with reduced processing requirements and/or higher processing speed, in mobile device such as mobile phones having display, camera and orientation sensor.

公开(公告)号：US11315308B2

公开(公告)日：2022-04-26

申请号：US17129340

申请日：2020-12-21

Applicant: Apple Inc.

Inventor： Lejing Wang ,  Peter Meier ,  Stefan Misslinger

IPC: G06T15/20 ,  G06T15/06 ,  G06T19/00

Abstract: A method for representing virtual information in a view of a real environment is provided that includes: providing a system setup including at least one display device, wherein the system setup is adapted for blending in virtual information on the display device in at least part of the view, determining a position and orientation of a viewing point relative to at least one component of the real environment, providing a geometry model of the real environment, providing at least one item of virtual information and a position of the at least one item of virtual information, determining whether the position of the item of virtual information is inside a 2D or 3D geometrical shape, determining a criterion which is indicative of whether the built-in real object is at least partially visible or non-visible in the view of the real environment, and blending in the at least one item of virtual information on the display device in at least part of the view of the real environment.

公开(公告)号：US09922446B2

公开(公告)日：2018-03-20

申请号：US14654395

申请日：2012-12-21

Applicant: Apple Inc.

Inventor： Lejing Wang ,  Peter Meier ,  Stefan Misslinger

IPC: G09G5/00 ,  G06T15/20 ,  G06T15/06 ,  G06T19/00

CPC classification number: G06T15/20 ,  G06T15/06 ,  G06T19/006

Abstract: A method for representing virtual information in a view of a real environment is provided that includes: providing a system setup including at least one display device, wherein the system setup is adapted for blending in virtual information on the display device in at least part of the view, determining a position and orientation of a viewing point relative to at least one component of the real environment, providing a geometry model of the real environment, providing at least one item of virtual information and a position of the at least one item of virtual information, determining whether the position of the item of virtual information is inside a 2D or 3D geometrical shape, determining a criterion which is indicative of whether the built-in real object is at least partially visible or non-visible in the view of the real environment, and blending in the at least one item of virtual information on the display device in at least part of the view of the real environment.

公开(公告)号：US09646422B2

公开(公告)日：2017-05-09

申请号：US15024800

申请日：2013-09-24

Applicant: Apple Inc.

Inventor： Anton Fedosov ,  Stefan Misslinger ,  Peter Meier

IPC: H04M1/00 ,  G06T19/00

CPC classification number: G06T19/006 ,  G01C21/3611 ,  G01C21/3638 ,  G01C21/3679 ,  G06F3/011 ,  G06T15/20 ,  H04L29/06034

Abstract: There is disclosed a method and mobile device for displaying points of interest in a view of a real environment displayed on a screen of the mobile device with a functionality for interaction with a user, which comprises the steps of: capturing an image of the real environment or a part of the real environment using a camera, determining at least one point of interest related to the real environment, determining an image position of the at least one point of interest in the image, displaying at least part of the image on at least part of the screen, overlaying a computer-generated indicator with the at least part of the image on the screen at a screen position according to the image position of the at least one point of interest, displaying a computer-generated virtual object related to the at least one point of interest on the screen at a screen position determined according to the screen position of the computer-generated indicator and which is adjacent to a bottom edge of the screen, displaying a visually perceivable relation indication indicative of a relation between the computer-generated virtual object and the computer-generated indicator. The mobile device may perform an action related to the at least one point of interest if at least part of the computer-generated virtual object displayed on the screen is touched.

公开(公告)号：US10217288B2

公开(公告)日：2019-02-26

申请号：US15480142

申请日：2017-04-05

Applicant: Apple Inc.

Inventor： Anton Fedosov ,  Stefan Misslinger ,  Peter Meier

IPC: G06F3/01 ,  G01C21/36 ,  G06T19/00 ,  H04L29/06 ,  G06T15/20

Abstract: There is disclosed a method and mobile device for displaying points of interest in a view of a real environment displayed on a screen of the mobile device with a functionality for interaction with a user, which comprises the steps of: capturing an image of the real environment or a part of the real environment using a camera, determining at least one point of interest related to the real environment, determining an image position of the at least one point of interest in the image, displaying at least part of the image on at least part of the screen, overlaying a computer-generated indicator with the at least part of the image on the screen at a screen position according to the image position of the at least one point of interest, displaying a computer-generated virtual object related to the at least one point of interest on the screen at a screen position determined according to the screen position of the computer-generated indicator and which is adjacent to a bottom edge of the screen, displaying a visually perceivable relation indication indicative of a relation between the computer-generated virtual object and the computer-generated indicator. The mobile device may perform an action related to the at least one point of interest if at least part of the computer-generated virtual object displayed on the screen is touched.

公开(公告)号：US20170213393A1

公开(公告)日：2017-07-27

申请号：US15480142

申请日：2017-04-05

Applicant: Apple Inc.

Inventor： Anton Fedosov ,  Stefan Misslinger ,  Peter Meier

IPC: G06T19/00 ,  G06T15/20 ,  H04L29/06 ,  G01C21/36 ,  G06F3/01

CPC classification number: G06T19/006 ,  G01C21/3611 ,  G01C21/3638 ,  G01C21/3679 ,  G06F3/011 ,  G06T15/20 ,  H04L29/06034

Abstract: There is disclosed a method and mobile device for displaying points of interest in a view of a real environment displayed on a screen of the mobile device with a functionality for interaction with a user, which comprises the steps of: capturing an image of the real environment or a part of the real environment using a camera, determining at least one point of interest related to the real environment, determining an image position of the at least one point of interest in the image, displaying at least part of the image on at least part of the screen, overlaying a computer-generated indicator with the at least part of the image on the screen at a screen position according to the image position of the at least one point of interest, displaying a computer-generated virtual object related to the at least one point of interest on the screen at a screen position determined according to the screen position of the computer-generated indicator and which is adjacent to a bottom edge of the screen, displaying a visually perceivable relation indication indicative of a relation between the computer-generated virtual object and the computer-generated indicator. The mobile device may perform an action related to the at least one point of interest if at least part of the computer-generated virtual object displayed on the screen is touched.

公开(公告)号：US20210183136A1

公开(公告)日：2021-06-17

申请号：US17129340

申请日：2020-12-21

Applicant: Apple Inc.

Inventor： Lejing Wang ,  Peter Meier ,  Stefan Misslinger

IPC: G06T15/20 ,  G06T15/06 ,  G06T19/00

Abstract: A method for representing virtual information in a view of a real environment is provided that includes: providing a system setup including at least one display device, wherein the system setup is adapted for blending in virtual information on the display device in at least part of the view, determining a position and orientation of a viewing point relative to at least one component of the real environment, providing a geometry model of the real environment, providing at least one item of virtual information and a position of the at least one item of virtual information, determining whether the position of the item of virtual information is inside a 2D or 3D geometrical shape, determining a criterion which is indicative of whether the built-in real object is at least partially visible or non-visible in the view of the real environment, and blending in the at least one item of virtual information on the display device in at least part of the view of the real environment.

公开(公告)号：US10820795B1

公开(公告)日：2020-11-03

申请号：US16015683

申请日：2018-06-22

Applicant: Apple Inc.

Inventor： Thibaut Weise ,  Justin D. Stoyles ,  Michael Kuhn ,  Reinhard Klapfer ,  Stefan Misslinger

IPC: A61B3/11 ,  G02C13/00 ,  G02B27/01 ,  G06K9/20 ,  G02B7/02

Abstract: In one implementation, a method includes: determining an interpupillary distance (IPD) measurement for a user based on a function of depth data obtained by the depth sensor and image data obtained by the image sensor; and calibrating a head-mounted device (HMD) provided to deliver augmented reality/virtual reality (AR/VR) content by setting one or more presentation parameters of the HMD based on the IPD measurement in order to tailor one or more AR/VR displays of the HMD to a field-of-view of the user.

公开(公告)号：US20200242846A1

公开(公告)日：2020-07-30

申请号：US16843462

申请日：2020-04-08

Applicant: Apple Inc.

Inventor： Peter Meier ,  Stefan Misslinger ,  Anton Fedosov

IPC: G06T19/00 ,  G06T15/20 ,  G06F3/01

Abstract: The invention relates to a method for ergonomically representing virtual information in a real environment, including the following steps: providing at least one view of a real environment and of a system setup for blending in virtual information for superimposing with the real environment in at least part of the view, the system setup comprising at least one display device, ascertaining a position and orientation of at least one part of the system setup relative to at least one component of the real environment, subdividing at least part of the view of the real environment into a plurality of regions comprising a first region and a second region, with objects of the real environment within the first region being placed closer to the system setup than objects of the real environment within the second region, and blending in at least one item of virtual information on the display device in at least part of the view of the real environment, considering the position and orientation of said at least one part of the system setup, wherein the virtual information is shown differently in the first region than in the second region with respect to the type of blending in the view of the real environment.