公开(公告)号：US11244111B2

公开(公告)日：2022-02-08

申请号：US16668333

申请日：2019-10-30

Applicant: salesforce.com, inc.

Inventor： Jiasen Lu ,  Caiming Xiong ,  Richard Socher

IPC: G06K9/00 ,  G06F40/274 ,  G06K9/62 ,  G06K9/46 ,  G06N3/04 ,  G06N3/08 ,  G06F40/30 ,  G06F40/169 ,  G06K9/48 ,  G06K9/66

Abstract: The technology disclosed presents a novel spatial attention model that uses current hidden state information of a decoder long short-term memory (LSTM) to guide attention and to extract spatial image features for use in image captioning. The technology disclosed also presents a novel adaptive attention model for image captioning that mixes visual information from a convolutional neural network (CNN) and linguistic information from an LSTM. At each timestep, the adaptive attention model automatically decides how heavily to rely on the image, as opposed to the linguistic model, to emit the next caption word. The technology disclosed further adds a new auxiliary sentinel gate to an LSTM architecture and produces a sentinel LSTM (Sn-LSTM). The sentinel gate produces a visual sentinel at each timestep, which is an additional representation, derived from the LSTM's memory, of long and short term visual and linguistic information.

公开(公告)号：US10558750B2

公开(公告)日：2020-02-11

申请号：US15817153

申请日：2017-11-17

Applicant: salesforce.com, inc.

Inventor： Jiasen Lu ,  Caiming Xiong ,  Richard Socher

IPC: G06F17/27 ,  G06K9/00 ,  G06K9/62 ,  G06K9/46 ,  G06F17/24 ,  G06K9/48 ,  G06K9/66 ,  G06N3/08

Abstract: The technology disclosed presents a novel spatial attention model that uses current hidden state information of a decoder long short-term memory (LSTM) to guide attention and to extract spatial image features for use in image captioning. The technology disclosed also presents a novel adaptive attention model for image captioning that mixes visual information from a convolutional neural network (CNN) and linguistic information from an LSTM. At each timestep, the adaptive attention model automatically decides how heavily to rely on the image, as opposed to the linguistic model, to emit the next caption word. The technology disclosed further adds a new auxiliary sentinel gate to an LSTM architecture and produces a sentinel LSTM (Sn-LSTM). The sentinel gate produces a visual sentinel at each timestep, which is an additional representation, derived from the LSTM's memory, of long and short term visual and linguistic information.

公开(公告)号：US10565306B2

公开(公告)日：2020-02-18

申请号：US15817165

申请日：2017-11-18

Applicant: salesforce.com, inc.

Inventor： Jiasen Lu ,  Caiming Xiong ,  Richard Socher

IPC: G06K9/66 ,  G06F17/27 ,  G06K9/00 ,  G06K9/62 ,  G06K9/46 ,  G06F17/24 ,  G06K9/48 ,  G06N3/08

Abstract: The technology disclosed presents a novel spatial attention model that uses current hidden state information of a decoder long short-term memory (LSTM) to guide attention and to extract spatial image features for use in image captioning. The technology disclosed also presents a novel adaptive attention model for image captioning that mixes visual information from a convolutional neural network (CNN) and linguistic information from an LSTM. At each timestep, the adaptive attention model automatically decides how heavily to rely on the image, as opposed to the linguistic model, to emit the next caption word. The technology disclosed further adds a new auxiliary sentinel gate to an LSTM architecture and produces a sentinel LSTM (Sn-LSTM). The sentinel gate produces a visual sentinel at each timestep, which is an additional representation, derived from the LSTM's memory, of long and short term visual and linguistic information.

公开(公告)号：US10565305B2

公开(公告)日：2020-02-18

申请号：US15817161

申请日：2017-11-17

Applicant: salesforce.com, inc.

Inventor： Jiasen Lu ,  Caiming Xiong ,  Richard Socher

IPC: G06K9/00 ,  G06F17/27 ,  G06K9/62 ,  G06K9/46 ,  G06F17/24 ,  G06K9/48 ,  G06K9/66 ,  G06N3/08 ,  G06N3/04

Abstract: The technology disclosed presents a novel spatial attention model that uses current hidden state information of a decoder long short-term memory (LSTM) to guide attention and to extract spatial image features for use in image captioning. The technology disclosed also presents a novel adaptive attention model for image captioning that mixes visual information from a convolutional neural network (CNN) and linguistic information from an LSTM. At each timestep, the adaptive attention model automatically decides how heavily to rely on the image, as opposed to the linguistic model, to emit the next caption word. The technology disclosed further adds a new auxiliary sentinel gate to an LSTM architecture and produces a sentinel LSTM (Sn-LSTM). The sentinel gate produces a visual sentinel at each timestep, which is an additional representation, derived from the LSTM's memory, of long and short term visual and linguistic information.