公开(公告)号：US20200084465A1

公开(公告)日：2020-03-12

申请号：US16687405

申请日：2019-11-18

Applicant: Salesforce.com, inc.

Inventor： Yingbo ZHOU ,  Luowei ZHOU ,  Caiming XIONG ,  Richard SOCHER

IPC: H04N19/46 ,  H04N19/132 ,  H04N19/126 ,  H04N19/33 ,  H04N21/81 ,  H04N19/187 ,  H04N19/60 ,  H04N19/44

Abstract: Systems and methods for dense captioning of a video include a multi-layer encoder stack configured to receive information extracted from a plurality of video frames, a proposal decoder coupled to the encoder stack and configured to receive one or more outputs from the encoder stack, a masking unit configured to mask the one or more outputs from the encoder stack according to one or more outputs from the proposal decoder, and a decoder stack coupled to the masking unit and configured to receive the masked one or more outputs from the encoder stack. Generating the dense captioning based on one or more outputs of the decoder stack. In some embodiments, the one or more outputs from the proposal decoder include a differentiable mask. In some embodiments, during training, error in the dense captioning is back propagated to the decoder stack, the encoder stack, and the proposal decoder.

公开(公告)号：US20190149834A1

公开(公告)日：2019-05-16

申请号：US15874515

申请日：2018-01-18

Applicant: salesforce.com, inc.

Inventor： Yingbo Zhou ,  Luowei ZHOU ,  Caiming XIONG ,  Richard SOCHER

IPC: H04N19/46 ,  H04N19/44 ,  H04N19/60 ,  H04N19/187 ,  H04N19/132 ,  H04N19/33 ,  H04N19/126 ,  H04N21/488 ,  H04N21/81

CPC classification number: H04N19/46 ,  H04N19/126 ,  H04N19/132 ,  H04N19/187 ,  H04N19/33 ,  H04N19/44 ,  H04N19/60 ,  H04N21/4884 ,  H04N21/8126

Abstract: Systems and methods for dense captioning of a video include a multi-layer encoder stack configured to receive information extracted from a plurality of video frames, a proposal decoder coupled to the encoder stack and configured to receive one or more outputs from the encoder stack, a masking unit configured to mask the one or more outputs from the encoder stack according to one or more outputs from the proposal decoder, and a decoder stack coupled to the masking unit and configured to receive the masked one or more outputs from the encoder stack. Generating the dense captioning based on one or more outputs of the decoder stack. In some embodiments, the one or more outputs from the proposal decoder include a differentiable mask. In some embodiments, during training, error in the dense captioning is back propagated to the decoder stack, the encoder stack, and the proposal decoder.