公开(公告)号：US20210150366A1

公开(公告)日：2021-05-20

申请号：US16877333

申请日：2020-05-18

Applicant: salesforce.com, inc.

Inventor： Govardana Sachithanandam Ramachandran ,  Ka Chun Au ,  Shashank Harinath ,  Wenhao Liu ,  Alexis Roos ,  Caiming Xiong

IPC: G06N3/08 ,  G06N20/00 ,  G06K9/62

Abstract: An embodiment proposed herein uses sparsification techniques to train the neural network with a high feature dimension that may yield desirable in-domain detection accuracy but may prune away dimensions in the output that are less important. Specifically, a sparsification vector is generated based on Gaussian distribution (or other probabilistic distribution) and is used to multiply with the higher dimension output to reduce the number of feature dimensions. The pruned output may be then used for the neural network to learn the sparsification vector. In this way, out-of-distribution detection accuracy can be improved.

公开(公告)号：US20210150340A1

公开(公告)日：2021-05-20

申请号：US16877339

申请日：2020-05-18

Applicant: salesforce.com, inc.

Inventor： Wenhao Liu ,  Ka Chun Au ,  Shashank Harinath ,  Bryan McCann ,  Govardana Sachithanandam Ramachandran ,  Alexis Roos ,  Caiming Xiong

IPC: G06N3/08 ,  G06F40/40 ,  G06N3/04

Abstract: Embodiments described herein provides a training mechanism that transfers the knowledge from a trained BERT model into a much smaller model to approximate the behavior of BERT. Specifically, the BERT model may be treated as a teacher model, and a much smaller student model may be trained using the same inputs to the teacher model and the output from the teacher model. In this way, the student model can be trained within a much shorter time than the BERT teacher model, but with comparable performance with BERT.

公开(公告)号：US20240078389A1

公开(公告)日：2024-03-07

申请号：US18505708

申请日：2023-11-09

Applicant: salesforce.com, inc.

Inventor： Ehsan Hosseini-Asl ,  Wenhao Liu

IPC: G06F40/30 ,  G06F40/284 ,  G06N3/04 ,  G06N3/08

CPC classification number: G06F40/30 ,  G06F40/284 ,  G06N3/04 ,  G06N3/08

Abstract: Sentiment analysis is a task in natural language processing. The embodiments are directed to using a generative language model to extract an aspect term, aspect category and their corresponding polarities. The generative language model may be trained as a single, joint, and multi-task model. The single-task generative language model determines a term polarity from the aspect term in the sentence or a category polarity from an aspect category in the sentence. The joint-task generative language model determines both the aspect term and the term polarity or the aspect category and the category polarity. The multi-task generative language model determines the aspect term, term polarity, aspect category and category polarity of the sentence.

公开(公告)号：US11922303B2

公开(公告)日：2024-03-05

申请号：US16877339

申请日：2020-05-18

Applicant: Salesforce.com, Inc.

Inventor： Wenhao Liu ,  Ka Chun Au ,  Shashank Harinath ,  Bryan McCann ,  Govardana Sachithanandam Ramachandran ,  Alexis Roos ,  Caiming Xiong

IPC: G06N3/00 ,  G06F40/40 ,  G06N3/045 ,  G06N3/08

CPC classification number: G06N3/08 ,  G06F40/40 ,  G06N3/045

Abstract: Embodiments described herein provides a training mechanism that transfers the knowledge from a trained BERT model into a much smaller model to approximate the behavior of BERT. Specifically, the BERT model may be treated as a teacher model, and a much smaller student model may be trained using the same inputs to the teacher model and the output from the teacher model. In this way, the student model can be trained within a much shorter time than the BERT teacher model, but with comparable performance with BERT.

公开(公告)号：US20230120940A1

公开(公告)日：2023-04-20

申请号：US17589693

申请日：2022-01-31

Applicant: salesforce.com, inc.

Inventor： Liang Qiu ,  Chien-Sheng Wu ,  Wenhao Liu ,  Caiming Xiong

IPC: G10L15/06 ,  G10L15/183 ,  G10L15/05

Abstract: Embodiments described herein propose an approach for unsupervised structure extraction in task-oriented dialogues. Specifically, a Slot Boundary Detection (SBD) module is adopted, for which utterances from training domains are tagged with the conventional BIO schema but without the slot names. A transformer-based classifier is trained to detect the boundary of potential slot tokens in the test domain. Next, while the state number is usually unknown, it is more reasonable to assume the slot number is given when analyzing a dialogue system. The detected tokens are clustered into the number of slot of groups. Finally, the dialogue state is represented with a vector recording the modification times of every slot. The slot values are then tracked through each dialogue session in the corpus and label utterances with their dialogue states accordingly. The semantic structure is portrayed by computing the transition frequencies among the unique states.

公开(公告)号：US11544470B2

公开(公告)日：2023-01-03

申请号：US17005316

申请日：2020-08-28

Applicant: salesforce.com, inc.

Inventor： Jianguo Zhang ,  Kazuma Hashimoto ,  Chien-Sheng Wu ,  Wenhao Liu ,  Richard Socher ,  Caiming Xiong

IPC: G06F17/00 ,  G06F40/30 ,  G10L15/16

Abstract: An online system allows user interactions using natural language expressions. The online system uses a machine learning based model to infer an intent represented by a user expression. The machine learning based model takes as input a user expression and an example expression to compute a score indicating whether the user expression matches the example expression. Based on the scores, the intent inference module determines a most applicable intent for the expression. The online system determines a confidence threshold such that user expressions indicating a high confidence are assigned the most applicable intent and user expressions indicating a low confidence are assigned an out-of-scope intent. The online system encodes the example expressions using the machine learning based model. The online system may compare an encoded user expression with encoded example expressions to identify a subset of example expressions used to determine the most applicable intent.

公开(公告)号：US20220101844A1

公开(公告)日：2022-03-31

申请号：US17037556

申请日：2020-09-29

Applicant: salesforce.com, inc.

Inventor： Xinyi Yang ,  Tian Xie ,  Caiming Xiong ,  Wenhao Liu ,  Huan Wang ,  Kazuma Hashimoto ,  Yingbo Zhou ,  Xugang Ye ,  Jin Qu ,  Feihong Wu

IPC: G10L15/22 ,  G10L15/26 ,  G10L15/16 ,  G10L15/30

Abstract: A conversation engine performs conversations with users using chatbots customized for performing a set of tasks that can be performed using an online system. The conversation engine loads a chatbot configuration that specifies the behavior of a chatbot including the tasks that can be performed by the chatbot, the types of entities relevant to each task, and so on. The conversation may be voice based and use natural language. The conversation engine may load different chatbot configurations to implement different chatbots. The conversation engine receives a conversation engine configuration that specifies the behavior of the conversation engine across chatbots. The system may be a multi-tenant system that allows customization of the chatbots for each tenant.

公开(公告)号：US20210375269A1

公开(公告)日：2021-12-02

申请号：US16999426

申请日：2020-08-21

Applicant: salesforce.com, inc.

Inventor： Semih Yavuz ,  Kazuma Hashimoto ,  Wenhao Liu ,  Nitish Shirish Keskar ,  Richard Socher ,  Caiming Xiong

IPC: G10L15/183 ,  G06N20/00 ,  G10L15/06 ,  G06F17/18

Abstract: Embodiments described herein utilize pre-trained masked language models as the backbone for dialogue act tagging and provide cross-domain generalization of the resulting dialogue acting taggers. For example, a pre-trained MASK token of BERT model may be used as a controllable mechanism for augmenting text input, e.g., generating tags for an input of unlabeled dialogue history. The pre-trained MASK model can be trained with semi-supervised learning, e.g., using multiple objectives from supervised tagging loss, masked tagging loss, masked language model loss, and/or a disagreement loss.