公开(公告)号：US10747761B2

公开(公告)日：2020-08-18

申请号：US15885613

申请日：2018-01-31

Applicant: salesforce.com, inc.

Inventor： Victor Zhong ,  Caiming Xiong ,  Richard Socher

IPC: G06F16/2452 ,  G06N3/08 ,  G06N7/00 ,  G06N3/04 ,  G06N3/00 ,  G06F16/13 ,  G06F16/2457

Abstract: A computing system uses neural networks to translate natural language queries to database queries. The computing system uses a plurality of machine learning based models, each machine learning model for generating a portion of the database query. The machine learning models use an input representation generated based on terms of the input natural language query, a set of columns of the database schema, and the vocabulary of a database query language, for example, structured query language SQL. The plurality of machine learning based models may include an aggregation classifier model for determining an aggregation operator in the database query, a result column predictor model for determining the result columns of the database query, and a condition clause predictor model for determining the condition clause of the database query. The condition clause predictor is based on reinforcement learning.

公开(公告)号：US10558750B2

公开(公告)日：2020-02-11

申请号：US15817153

申请日：2017-11-17

Applicant: salesforce.com, inc.

Inventor： Jiasen Lu ,  Caiming Xiong ,  Richard Socher

IPC: G06F17/27 ,  G06K9/00 ,  G06K9/62 ,  G06K9/46 ,  G06F17/24 ,  G06K9/48 ,  G06K9/66 ,  G06N3/08

Abstract: The technology disclosed presents a novel spatial attention model that uses current hidden state information of a decoder long short-term memory (LSTM) to guide attention and to extract spatial image features for use in image captioning. The technology disclosed also presents a novel adaptive attention model for image captioning that mixes visual information from a convolutional neural network (CNN) and linguistic information from an LSTM. At each timestep, the adaptive attention model automatically decides how heavily to rely on the image, as opposed to the linguistic model, to emit the next caption word. The technology disclosed further adds a new auxiliary sentinel gate to an LSTM architecture and produces a sentinel LSTM (Sn-LSTM). The sentinel gate produces a visual sentinel at each timestep, which is an additional representation, derived from the LSTM's memory, of long and short term visual and linguistic information.

公开(公告)号：US10542270B2

公开(公告)日：2020-01-21

申请号：US15874515

申请日：2018-01-18

Applicant: salesforce.com, inc.

Inventor： Yingbo Zhou ,  Luowei Zhou ,  Caiming Xiong ,  Richard Socher

IPC: H04N7/12 ,  H04N11/12 ,  H04N19/46 ,  H04N19/44 ,  H04N19/60 ,  H04N19/187 ,  H04N21/81 ,  H04N19/33 ,  H04N19/126 ,  H04N21/488 ,  H04N19/132

Abstract: Systems and methods for dense captioning of a video include a multi-layer encoder stack configured to receive information extracted from a plurality of video frames, a proposal decoder coupled to the encoder stack and configured to receive one or more outputs from the encoder stack, a masking unit configured to mask the one or more outputs from the encoder stack according to one or more outputs from the proposal decoder, and a decoder stack coupled to the masking unit and configured to receive the masked one or more outputs from the encoder stack. Generating the dense captioning based on one or more outputs of the decoder stack. In some embodiments, the one or more outputs from the proposal decoder include a differentiable mask. In some embodiments, during training, error in the dense captioning is back propagated to the decoder stack, the encoder stack, and the proposal decoder.

公开(公告)号：US20190258939A1

公开(公告)日：2019-08-22

申请号：US15980207

申请日：2018-05-15

Applicant: salesforce.com, inc.

Inventor： Sewon Min ,  Victor Zhong ,  Caiming Xiong ,  Richard Socher

IPC: G06N3/08 ,  G06F17/27

Abstract: A natural language processing system that includes a sentence selector and a question answering module. The sentence selector receives a question and sentences that are associated with a context. For a question and each sentence, the sentence selector determines a score. A score represents whether the question is answerable with the sentence. Sentence selector then generates a minimum set of sentences from the scores associated with the question and sentences. The question answering module generates an answer for the question from the minimum set of sentences.

公开(公告)号：US20180336198A1

公开(公告)日：2018-11-22

申请号：US15885613

申请日：2018-01-31

Applicant: salesforce.com, inc.

Inventor： Victor Zhong ,  Caiming Xiong ,  Richard Socher

IPC: G06F17/30 ,  G06N3/08 ,  G06N7/00

Abstract: A computing system uses neural networks to translate natural language queries to database queries. The computing system uses a plurality of machine learning based models, each machine learning model for generating a portion of the database query. The machine learning models use an input representation generated based on terms of the input natural language query, a set of columns of the database schema, and the vocabulary of a database query language, for example, structured query language SQL. The plurality of machine learning based models may include an aggregation classifier model for determining an aggregation operator in the database query, a result column predictor model for determining the result columns of the database query, and a condition clause predictor model for determining the condition clause of the database query. The condition clause predictor is based on reinforcement learning.

公开(公告)号：US11749264B2

公开(公告)日：2023-09-05

申请号：US17088206

申请日：2020-11-03

Applicant: salesforce.com, inc.

Inventor： Chien-Sheng Wu ,  Chu Hong Hoi ,  Richard Socher ,  Caiming Xiong

IPC: G10L15/18 ,  G10L15/06

CPC classification number: G10L15/1815 ,  G10L15/063 ,  G10L15/1822

Abstract: Embodiments described herein provide methods and systems for training task-oriented dialogue (TOD) language models. In some embodiments, a TOD language model may receive a TOD dataset including a plurality of dialogues and a model input sequence may be generated from the dialogues using a first token prefixed to each user utterance and a second token prefixed to each system response of the dialogues. In some embodiments, the first token or the second token may be randomly replaced with a mask token to generate a masked training sequence and a masked language modeling (MLM) loss may be computed using the masked training sequence. In some embodiments, the TOD language model may be updated based on the MLM loss.

公开(公告)号：US11687588B2

公开(公告)日：2023-06-27

申请号：US16531343

申请日：2019-08-05

Applicant: salesforce.com, inc.

Inventor： Mingfei Gao ,  Richard Socher ,  Caiming Xiong

IPC: G06F16/735 ,  G06F16/73 ,  G06V10/82 ,  G06F16/74 ,  G06V20/40 ,  G06F17/10 ,  G06N3/08 ,  G06F40/47 ,  G06F18/21 ,  G06V10/44

CPC classification number: G06F16/735 ,  G06F16/73 ,  G06F17/10 ,  G06F18/2185 ,  G06F40/47 ,  G06N3/08 ,  G06V10/82 ,  G06V20/41 ,  G06V20/49 ,  G06V10/454 ,  G06V20/44 ,  G06V20/46

Abstract: Systems and methods are provided for weakly supervised natural language localization (WSNLL), for example, as implemented in a neural network or model. The WSNLL network is trained with long, untrimmed videos, i.e., videos that have not been temporally segmented or annotated. The WSNLL network or model defines or generates a video-sentence pair, which corresponds to a pairing of an untrimmed video with an input text sentence. According to some embodiments, the WSNLL network or model is implemented with a two-branch architecture, where one branch performs segment sentence alignment and the other one conducts segment selection. These methods and systems are specifically used to predict how a video proposal matches a text query using respective visual and text features.

公开(公告)号：US11631009B2

公开(公告)日：2023-04-18

申请号：US16051309

申请日：2018-07-31

Applicant: salesforce.com, inc.

Inventor： Xi Victoria Lin ,  Caiming Xiong ,  Richard Socher

IPC: G06N20/00 ,  G06N5/04 ,  G06N3/04 ,  G06N3/08 ,  G06F16/903 ,  G06F16/901

Abstract: Approaches for multi-hop knowledge graph reasoning with reward shaping include a system and method of training a system to search relational paths in a knowledge graph. The method includes identifying, using an reasoning module, a plurality of first outgoing links from a current node in a knowledge graph, masking, using the reasoning module, one or more links from the plurality of first outgoing links to form a plurality of second outgoing links, rewarding the reasoning module with a reward of one when a node corresponding to an observed answer is reached, and rewarding the reasoning module with a reward identified by a reward shaping network when a node not corresponding to an observed answer is reached. In some embodiments, the reward shaping network is pre-trained.

公开(公告)号：US11599721B2

公开(公告)日：2023-03-07

申请号：US17002562

申请日：2020-08-25

Applicant: salesforce.com, inc.

Inventor： Shiva Kumar Pentyala ,  Mridul Gupta ,  Ankit Chadha ,  Indira Iyer ,  Richard Socher

IPC: G06F40/253 ,  G10L15/19 ,  G06F40/30

Abstract: A natural language processing system that trains task models for particular natural language tasks programmatically generates additional utterances for inclusion in the training set, based on the existing utterances in the training set and the existing state of a task model as generated from the original (non-augmented) training set. More specifically, the training augmentation module 220 identifies specific textual units of utterances and generates variants of the utterances based on those identified units. The identification is based on determined importances of the textual units to the output of the task model, as well as on task rules that correspond to the natural language task for which the task model is being generated. The generation of the additional utterances improves the quality of the task model without the expense of manual labeling of utterances for training set inclusion.

公开(公告)号：US11580359B2

公开(公告)日：2023-02-14

申请号：US16664508

申请日：2019-10-25

Applicant: salesforce.com, inc.

Inventor： Stephen Joseph Merity ,  Caiming Xiong ,  James Bradbury ,  Richard Socher

IPC: G06N3/04 ,  G06N3/084 ,  G06F40/284 ,  G06N3/08 ,  G06N7/00

Abstract: The technology disclosed provides a so-called “pointer sentinel mixture architecture” for neural network sequence models that has the ability to either reproduce a token from a recent context or produce a token from a predefined vocabulary. In one implementation, a pointer sentinel-LSTM architecture achieves state of the art language modeling performance of 70.9 perplexity on the Penn Treebank dataset, while using far fewer parameters than a standard softmax LSTM.