SEMI-SUPERVISED AND ROBUST MULTISPECTRAL VIDEO SEMANTIC SEGMENTATION SYSTEM

    公开(公告)号:US20250166339A1

    公开(公告)日:2025-05-22

    申请号:US18955003

    申请日:2024-11-21

    Abstract: A method includes generating a pair of features of a current frame by inputting the current frame into a first cross-collaborative consistency learning (C3L) model, the current frame comprising a red-green-blue (RGB) image and a thermal image; generating a pair of denoised features by inputting the of pair of features of the current frame and one or more pairs of features of past frames into a denoised memory read (DMR) model; generating an updated pair of denoised features by inputting the pair of denoised features into a second C3L model, the updated pair of denoised features comprising an updated RGB image feature and an updated thermal feature; and generating a segmentation mask by inputting the updated pair of features into a segmentation head.

    APPARATUS AND METHOD FOR SHARING AND PRUNING WEIGHTS FOR VISION AND LANGUAGE MODELS

    公开(公告)号:US20240119077A1

    公开(公告)日:2024-04-11

    申请号:US18368353

    申请日:2023-09-14

    CPC classification number: G06F16/334 G06F16/5846 G06N3/0985

    Abstract: A method of performing a multimodal tasks by using a multimodal model that includes a text encoder and a vision encoder, may include obtaining a text feature from the query via the text encoder; obtaining an image feature from the one or more input images via the vision encoder; and outputting a response to the query based on similarity between the text feature and the image feature, wherein weights vectors of the text encoder and the vision encoder are pruned and shared according to a sharing vector and a pruning vector that are generated by a hypernetwork, and wherein the hypernetwork and the multimodal model are jointly trained to minimize at least one of a difference between the weight vectors in the text encoder and the vision encoder, a difference between the weight vectors in different layers of the text encoder, and a number of parameters in the multimodal model.

Patent Agency Ranking