公开(公告)号：US20240282090A1

公开(公告)日：2024-08-22

申请号：US18516124

申请日：2023-11-21

Applicant: WUHAN UNIVERSITY

Inventor： Juan Liu ,  Na Zhang

IPC: G06V10/80 ,  A61B8/08 ,  G06T7/00 ,  G06V10/32 ,  G06V10/77 ,  G06V10/776 ,  G06V10/778 ,  G06V10/82 ,  G06V20/70 ,  G16H30/40 ,  G16H50/20

CPC classification number: G06V10/811 ,  A61B8/085 ,  A61B8/5223 ,  A61B8/5261 ,  G06T7/0012 ,  G06V10/32 ,  G06V10/7715 ,  G06V10/776 ,  G06V10/778 ,  G06V10/806 ,  G06V10/82 ,  G06V20/70 ,  G16H30/40 ,  G16H50/20 ,  G06T2207/10048 ,  G06T2207/10132 ,  G06T2207/20081 ,  G06T2207/30004 ,  G06V2201/03

Abstract: The present disclosure provides a multi-modal method for classifying a thyroid nodule based on ultrasound (US) and infrared thermal (IRT) images. Based on ultrasound and infrared thermal images and in combination with a multi-modal learning method, the present disclosure provides an adaptive multi-modal hybrid (AmmH) model which is composed of three parts: an intra-modal hybrid encoder (HIME), an adaptive cross-modal encoder (ACME), and a multilayer perceptron (MLP) head. The HIME is capable of modeling a global feature while extracting a local feature. The ACME is capable of customizing personalized modality-weights according to different cases and performing information interaction and fusion of inter-modal features. The MLP head classifies a fused feature obtained. The method enables the AmmH model to automatically classify a thyroid nodule of a subject based on ultrasound and infrared thermal images of the subject, providing a doctor with an objective and accurate classification result to assist diagnosis.