公开(公告)号：US20240119077A1

公开(公告)日：2024-04-11

申请号：US18368353

申请日：2023-09-14

Applicant: SAMSUNG ELECTRONICS CO., LTD.

Inventor： Shangqian GAO ,  Burak UZKENT ,  Yilin SHEN ,  Hongxia JIN

IPC: G06F16/33 ,  G06F16/583 ,  G06N3/0985

CPC classification number: G06F16/334 ,  G06F16/5846 ,  G06N3/0985

Abstract: A method of performing a multimodal tasks by using a multimodal model that includes a text encoder and a vision encoder, may include obtaining a text feature from the query via the text encoder; obtaining an image feature from the one or more input images via the vision encoder; and outputting a response to the query based on similarity between the text feature and the image feature, wherein weights vectors of the text encoder and the vision encoder are pruned and shared according to a sharing vector and a pruning vector that are generated by a hypernetwork, and wherein the hypernetwork and the multimodal model are jointly trained to minimize at least one of a difference between the weight vectors in the text encoder and the vision encoder, a difference between the weight vectors in different layers of the text encoder, and a number of parameters in the multimodal model.

公开(公告)号：US20230289590A1

公开(公告)日：2023-09-14

申请号：US17940709

申请日：2022-09-08

Applicant: SAMSUNG ELECTRONICS CO., LTD.

Inventor： Burak UZKENT ,  Vasili Ramanishka ,  Yilin Shen ,  Hongxia Jin

IPC: G06N3/08 ,  G06N3/063 ,  G10L25/54 ,  G10L15/26

CPC classification number: G06N3/08 ,  G06N3/063 ,  G10L25/54 ,  G10L15/26

Abstract: A method of training a model includes configuring a first transformer for visual learning with a first set of weights, configuring a second transformer for textual learning with a second set of weights, adjusting at least the second set of weights based on minimizing a weight difference between the first set of weights and the second set of weights, replacing the first set of weights for the first transformer with the adjusted second set of weights, and updating the first transformer based on the adjusted second set of weights.

公开(公告)号：US20230075862A1

公开(公告)日：2023-03-09

申请号：US17899118

申请日：2022-08-30

Applicant: SAMSUNG ELECTRONICS CO., LTD.

Inventor： Burak UZKENT ,  Vasili Ramanishka ,  Yilin Shen ,  Hongxia Jin

IPC: G06V10/82 ,  G06V10/86 ,  G06V10/75 ,  G06V10/94 ,  G06V10/56 ,  G06F16/53

Abstract: A method of training a neural network model includes generating a positive image based on an original image, generating a positive text corresponding to the positive image based on an original text corresponding to the original image, the positive text referring to an object in the positive image, constructing a positive image-text pair for the object based on the positive image and the positive text, constructing a negative image-text pair for the object based on the original image and a negative text, the negative text not referring to the object, training the neural network model based on the positive image-text pair and the negative image-text pair to output features representing an input image-text pair, and identifying the object in the original image based on the features representing the input image-text pair.