公开(公告)号：US20220035155A1

公开(公告)日：2022-02-03

申请号：US17381466

申请日：2021-07-21

Applicant: Sony Interactive Entertainment Inc.

Inventor： Nigel John Williams ,  Patrick John Connor

IPC: G02B27/00 ,  G02B27/01 ,  G06F3/01

Abstract: Apparatus comprises an image processor configured to provide output video images to a head mountable display, HMD, having one or more display elements to display video images to a wearer of the HMD, for display by the one or more display elements in response to input video images, in which each output video image corresponds to a respective input video image; in which: the image processor comprises a detector configured to detect whether an input image brightness at image locations in the input video images exceeds a threshold image brightness; and the image processor is configured to vary a relationship between the display properties at a given image location in a given output video image and the input image display properties at the given image location in the corresponding input video image in response to a detection that image brightness at the given image location in one or more input video images preceding that corresponding input video image exceeded the threshold image brightness.

公开(公告)号：US20210183146A1

公开(公告)日：2021-06-17

申请号：US17114596

申请日：2020-12-08

Applicant: Sony Interactive Entertainment Inc.

Inventor： Fabio Cappello ,  Maurizio Cerrato ,  Rosario Leonardi ,  Gregory James Bedwell ,  Nigel John Williams

IPC: G06T19/00 ,  H04N13/282 ,  H04N13/279 ,  H04N13/117

Abstract: A system for modifying data representing a virtual environment includes: an environment navigation unit operable to control navigation within the virtual environment to generate one or more viewpoints within the virtual environment, an environment identification unit operable to identify one or more aspects of the geometry of the virtual environment visible in the one or more viewpoints, a geometry evaluation unit operable to evaluate the visibility of one or more aspects of the geometry based upon the identification for each of one or more viewpoints, and a data modification unit operable to modify one or more elements of data representing the virtual environment.

公开(公告)号：US11710256B2

公开(公告)日：2023-07-25

申请号：US17007174

申请日：2020-08-31

Applicant: Sony Interactive Entertainment Inc.

Inventor： Nigel John Williams ,  Andrew William Walker

IPC: G06T7/80 ,  G06T7/593 ,  H04N13/106 ,  G06F1/03 ,  G06T5/50 ,  H04N13/111 ,  H04N13/00

CPC classification number: G06T7/80 ,  G06F1/03 ,  G06T5/50 ,  G06T7/593 ,  H04N13/106 ,  H04N13/111 ,  G06T2207/20021 ,  G06T2207/20216 ,  H04N2013/0081

Abstract: A method of generating a 3D reconstruction of a scene, the scene comprising a plurality of cameras positioned around the scene, comprises: obtaining the extrinsics and intrinsics of a virtual camera within a scene; accessing a data structure so as to determine a camera pair that is to be used in reconstructing the scene from the viewpoint of the virtual camera; wherein the data structure defines a voxel representation of the scene, the voxel representation comprising a plurality of voxels, at least some of the voxel surfaces being associated with respective camera pair identifiers; wherein each camera pair identifier associated with a respective voxel surface corresponds to a camera pair that has been identified as being suitable for obtaining depth data for the part of the scene within that voxel and for which the averaged pose of the camera pair is oriented towards the voxel surface; identifying, based on the obtained extrinsics and intrinsics of the virtual camera, at least one voxel that is within the field of view of the virtual camera and a corresponding voxel surface that is oriented towards the virtual camera; identifying, based on the accessed data structure, at least one camera pair that is suitable for reconstructing the scene from the viewpoint of the virtual camera, and generating a reconstruction of the scene from the viewpoint of the virtual camera based on the images captured by the cameras in the identified at least one camera pair.

公开(公告)号：US20230215047A1

公开(公告)日：2023-07-06

申请号：US18182669

申请日：2023-03-13

Applicant: Sony Interactive Entertainment Inc.

Inventor： Nigel John Williams ,  Andrew William Walker

IPC: G06T15/20 ,  H04N13/111 ,  G06T17/00 ,  G06F3/01

CPC classification number: G06T15/205 ,  G06F3/012 ,  G06T17/00 ,  H04N13/111 ,  H04N2013/0081

Abstract: Methods and/or apparatus provide for obtaining intrinsics and extrinsics of a plurality of cameras arranged to capture a scene and based thereon, generate a voxel representation of the scene; obtaining extrinsics and intrinsics of a virtual camera; mapping the extrinsics and intrinsics of the virtual camera to at least one voxel surface defined in a data structure and to identify at least one camera pair suitable for obtaining depth data for that voxel surface based on the corresponding camera pair identifier; and receiving an input from the mapping unit, and in response obtaining images captured by the cameras in the at least one camera pair identified as being suitable for reconstructing the scene from the viewpoint of the virtual camera, and to generate an image of the scene from the viewpoint of the virtual camera based on the obtained images.

公开(公告)号：US20230071832A1

公开(公告)日：2023-03-09

申请号：US18055511

申请日：2022-11-15

Applicant: Sony Interactive Entertainment Inc.

Inventor： Fabio Cappello ,  Maurizio Cerrato ,  Rosario Leonardi ,  Gregory James Bedwell ,  Nigel John Williams

IPC: G06T19/00 ,  H04N13/279 ,  H04N13/117 ,  H04N13/282

Abstract: A system for improving a discovery process for a virtual environment, the system comprising an environment discovery unit operable to perform a discovery process comprising navigation of the virtual environment and identification of one or more aspects of the virtual environment, a performance analysis unit operable to evaluate the effectiveness of the discovery process, and a discovery update unit operable to modify future operation of the environment discovery unit in dependence upon the evaluation.

公开(公告)号：US20210241495A1

公开(公告)日：2021-08-05

申请号：US17268856

申请日：2019-08-06

Applicant: Sony Interactive Entertainment Inc.

Inventor： Fabio Cappello ,  Nigel John Williams

IPC: G06T7/90 ,  G06T7/50 ,  G06T15/20

Abstract: A method of reconstructing colour and depth information of a scene includes receiving a colour image of a scene and obtaining depth information of the scene. The colour and depth images are used to generate a point cloud, which is then projected to an alternative viewpoint and converted to sparse colour and depth images. Colour information is then estimated for at least some parts of the sparse colour image, resulting in a reconstructed colour image. The reconstructed colour image is used with the existing depth information to estimate depth information for the sparse depth image. In this way, colour and depth information of the scene can be estimated, and used to generate colour and depth images of the scene from a desired viewpoint. A corresponding system for reconstructing colour and depth information is also provided.

公开(公告)号：US20210124996A1

公开(公告)日：2021-04-29

申请号：US17074827

申请日：2020-10-20

Applicant: Sony Interactive Entertainment Inc.

Inventor： Mark Jacobus Breugelmans ,  Oliver Hume ,  Fabio Cappello ,  Nigel John Williams

IPC: G06K9/62 ,  H04N1/44 ,  G06F21/62

Abstract: An encoding apparatus is provided. The apparatus comprises an input unit operable to obtain a plurality of training images, said training images being for use in training a machine learning model. The apparatus also comprises a label unit operable to obtain a class label associated with the training images; and a key unit operable to obtain a secret key for use in encoding the training images. The apparatus further comprises an image noise generator operable to generate, based on the obtained secret key, noise for introducing into the training images. The image noise generator is configured to generate noise that correlates with the class label associated with the training images such that a machine learning model subsequently trained with the modified training images learns to associate the introduced noise with the class label for those images. A corresponding decoding apparatus is also provided.

公开(公告)号：US20210074023A1

公开(公告)日：2021-03-11

申请号：US17007174

申请日：2020-08-31

Applicant: Sony Interactive Entertainment Inc.

Inventor： Nigel John Williams ,  Andrew William Walker

IPC: G06T7/80 ,  G06T7/593 ,  G06T5/50 ,  H04N13/106 ,  G06F1/03

Abstract: A method of generating a 3D reconstruction of a scene, the scene comprising a plurality of cameras positioned around the scene, comprises: obtaining the extrinsics and intrinsics of a virtual camera within a scene; accessing a data structure so as to determine a camera pair that is to be used in reconstructing the scene from the viewpoint of the virtual camera; wherein the data structure defines a voxel representation of the scene, the voxel representation comprising a plurality of voxels, at least some of the voxel surfaces being associated with respective camera pair identifiers; wherein each camera pair identifier associated with a respective voxel surface corresponds to a camera pair that has been identified as being suitable for obtaining depth data for the part of the scene within that voxel and for which the averaged pose of the camera pair is oriented towards the voxel surface; identifying, based on the obtained extrinsics and intrinsics of the virtual camera, at least one voxel that is within the field of view of the virtual camera and a corresponding voxel surface that is oriented towards the virtual camera; identifying, based on the accessed data structure, at least one camera pair that is suitable for reconstructing the scene from the viewpoint of the virtual camera, and generating a reconstruction of the scene from the viewpoint of the virtual camera based on the images captured by the cameras in the identified at least one camera pair.

公开(公告)号：US12056899B2

公开(公告)日：2024-08-06

申请号：US18182669

申请日：2023-03-13

Applicant: Sony Interactive Entertainment Inc.

Inventor： Nigel John Williams ,  Andrew William Walker

IPC: G06T7/80 ,  G06F1/03 ,  G06F3/01 ,  G06T5/50 ,  G06T7/593 ,  G06T15/20 ,  G06T17/00 ,  H04N13/00 ,  H04N13/106 ,  H04N13/111

CPC classification number: G06T7/80 ,  G06F1/03 ,  G06F3/012 ,  G06T5/50 ,  G06T7/593 ,  G06T15/205 ,  G06T17/00 ,  H04N13/106 ,  H04N13/111 ,  G06T2200/08 ,  G06T2207/20021 ,  G06T2207/20216 ,  H04N2013/0081

Abstract: Methods and/or apparatus provide for obtaining intrinsics and extrinsics of a plurality of cameras arranged to capture a scene and based thereon, generate a voxel representation of the scene; obtaining extrinsics and intrinsics of a virtual camera; mapping the extrinsics and intrinsics of the virtual camera to at least one voxel surface defined in a data structure and to identify at least one camera pair suitable for obtaining depth data for that voxel surface based on the corresponding camera pair identifier; and receiving an input from the mapping unit, and in response obtaining images captured by the cameras in the at least one camera pair identified as being suitable for reconstructing the scene from the viewpoint of the virtual camera, and to generate an image of the scene from the viewpoint of the virtual camera based on the obtained images.

公开(公告)号：US11941772B2

公开(公告)日：2024-03-26

申请号：US17273573

申请日：2019-09-04

Applicant: Sony Interactive Entertainment Inc.

Inventor： Fabio Cappello ,  Nigel John Williams ,  Rajeev Gupta

IPC: G06T19/20 ,  G06N3/04 ,  G06N3/08 ,  G06T7/10 ,  G06T7/50 ,  G06T7/90 ,  G06T15/04 ,  G06T17/00

CPC classification number: G06T19/20 ,  G06N3/04 ,  G06N3/08 ,  G06T7/10 ,  G06T7/50 ,  G06T7/90 ,  G06T15/04 ,  G06T17/00 ,  G06T2200/08 ,  G06T2207/10024 ,  G06T2207/20084 ,  G06T2207/30196 ,  G06T2219/2021

Abstract: A method of generating a three-dimensional (3D) reconstruction of a human includes obtaining at least one colour image and corresponding depth image of a scene comprising a human subject that is at least partially occluded by one or more items. The regions in the at least one colour image corresponding to a human subject are identified. A point cloud of the scene is generated based on the at least one colour image and corresponding depth image. One or more parameters of a parametric model are then adjusted based on the regions of the point cloud identified as corresponding to the human subject. The adjusted parametric model is then used to generate a 3D reconstruction of the human subject; where the parametric model provides an estimate for points in the point cloud corresponding to the occluded parts of the human subject. A corresponding system is also provided.