公开(公告)号：US10958925B2

公开(公告)日：2021-03-23

申请号：US16687405

申请日：2019-11-18

Applicant: salesforce.com, inc.

Inventor： Yingbo Zhou ,  Luowei Zhou ,  Caiming Xiong ,  Richard Socher

IPC: H04N19/46 ,  H04N19/44 ,  H04N19/60 ,  H04N19/187 ,  H04N21/81 ,  H04N19/33 ,  H04N19/126 ,  H04N19/132 ,  H04N21/488

Abstract: Systems and methods for dense captioning of a video include a multi-layer encoder stack configured to receive information extracted from a plurality of video frames, a proposal decoder coupled to the encoder stack and configured to receive one or more outputs from the encoder stack, a masking unit configured to mask the one or more outputs from the encoder stack according to one or more outputs from the proposal decoder, and a decoder stack coupled to the masking unit and configured to receive the masked one or more outputs from the encoder stack. Generating the dense captioning based on one or more outputs of the decoder stack. In some embodiments, the one or more outputs from the proposal decoder include a differentiable mask. In some embodiments, during training, error in the dense captioning is back propagated to the decoder stack, the encoder stack, and the proposal decoder.

公开(公告)号：US10542270B2

公开(公告)日：2020-01-21

申请号：US15874515

申请日：2018-01-18

Applicant: salesforce.com, inc.

Inventor： Yingbo Zhou ,  Luowei Zhou ,  Caiming Xiong ,  Richard Socher

IPC: H04N7/12 ,  H04N11/12 ,  H04N19/46 ,  H04N19/44 ,  H04N19/60 ,  H04N19/187 ,  H04N21/81 ,  H04N19/33 ,  H04N19/126 ,  H04N21/488 ,  H04N19/132

Abstract: Systems and methods for dense captioning of a video include a multi-layer encoder stack configured to receive information extracted from a plurality of video frames, a proposal decoder coupled to the encoder stack and configured to receive one or more outputs from the encoder stack, a masking unit configured to mask the one or more outputs from the encoder stack according to one or more outputs from the proposal decoder, and a decoder stack coupled to the masking unit and configured to receive the masked one or more outputs from the encoder stack. Generating the dense captioning based on one or more outputs of the decoder stack. In some embodiments, the one or more outputs from the proposal decoder include a differentiable mask. In some embodiments, during training, error in the dense captioning is back propagated to the decoder stack, the encoder stack, and the proposal decoder.