公开(公告)号：US10970486B2

公开(公告)日：2021-04-06

申请号：US16134956

申请日：2018-09-18

Applicant: salesforce.com, inc.

Inventor： Michael Machado ,  John Ball ,  Thomas Archie Cook, Jr. ,  Shashank Harinath ,  Roojuta Lalani ,  Zineb Laraki ,  Qingqing Liu ,  Mike Rosenbaum ,  Karl Ryszard Skucha ,  Jean-Marc Soumet ,  Manju Vijayakumar

IPC: G06F40/295 ,  G06F16/332 ,  G06F40/30 ,  G10L15/18 ,  G10L15/22 ,  G10L15/30

Abstract: Approaches to using unstructured input to update heterogeneous data stores include receiving unstructured text input, receiving a template for interpreting the unstructured text input, identifying, using an entity classifier, entities in the unstructured text input, identifying one or more potential parent entities from the identified entities based on the template, receiving a selection of a parent entity from the one or more potential parent entities, identifying one or more potential child entities from the identified entities based on the template and the selected parent entity, receiving a selection of a child entity from the one or more potential child entities, identifying an action item in the unstructured text input based on the identified entities and the template, determining, using an intent classifier, an intent of the action item, and updating a data store based on the determined intent, the identified entities, and the selected child entity.

公开(公告)号：US20230086302A1

公开(公告)日：2023-03-23

申请号：US17479748

申请日：2021-09-20

Applicant: salesforce.com, inc.

Inventor： Shilpa Bhagavath ,  Shubham Mehrotra ,  Abhishek Sharma ,  Shashank Harinath ,  Na Cheng ,  Zineb Laraki

IPC: G06F40/263 ,  G06F40/58 ,  G06K9/62

Abstract: A method that includes receiving an input at an interactive conversation service that uses an intent classification model. The method may further include generating, using an encoder model of the intent classification model, a set of output vectors corresponding to the input, where the encoder model is configured to determine a set of metrics corresponding to intent classifications. The method may further include determining, using an outlier detection model of the intent classification model, whether the input is in-domain or out-of-domain (OOD) based on a first vector of the set of output vectors satisfying a domain threshold relative to one or more of the intent classifications. The method may further include outputting, by the intent classification model, a second vector of the set of output vectors that indicates the set of metrics corresponding to the intent classifications or an indication that the input is OOD.

公开(公告)号：US11537899B2

公开(公告)日：2022-12-27

申请号：US16877333

申请日：2020-05-18

Applicant: salesforce.com, inc.

Inventor： Govardana Sachithanandam Ramachandran ,  Ka Chun Au ,  Shashank Harinath ,  Wenhao Liu ,  Alexis Roos ,  Caiming Xiong

IPC: G06N3/08 ,  G06N20/00 ,  G06K9/62 ,  G06N20/10 ,  G06F17/18 ,  G06V10/75

Abstract: An embodiment proposed herein uses sparsification techniques to train the neural network with a high feature dimension that may yield desirable in-domain detection accuracy but may prune away dimensions in the output that are less important. Specifically, a sparsification vector is generated based on Gaussian distribution (or other probabilistic distribution) and is used to multiply with the higher dimension output to reduce the number of feature dimensions. The pruned output may be then used for the neural network to learn the sparsification vector. In this way, out-of-distribution detection accuracy can be improved.

公开(公告)号：US20210150366A1

公开(公告)日：2021-05-20

申请号：US16877333

申请日：2020-05-18

Applicant: salesforce.com, inc.

Inventor： Govardana Sachithanandam Ramachandran ,  Ka Chun Au ,  Shashank Harinath ,  Wenhao Liu ,  Alexis Roos ,  Caiming Xiong

IPC: G06N3/08 ,  G06N20/00 ,  G06K9/62

Abstract: An embodiment proposed herein uses sparsification techniques to train the neural network with a high feature dimension that may yield desirable in-domain detection accuracy but may prune away dimensions in the output that are less important. Specifically, a sparsification vector is generated based on Gaussian distribution (or other probabilistic distribution) and is used to multiply with the higher dimension output to reduce the number of feature dimensions. The pruned output may be then used for the neural network to learn the sparsification vector. In this way, out-of-distribution detection accuracy can be improved.

公开(公告)号：US20210150340A1

公开(公告)日：2021-05-20

申请号：US16877339

申请日：2020-05-18

Applicant: salesforce.com, inc.

Inventor： Wenhao Liu ,  Ka Chun Au ,  Shashank Harinath ,  Bryan McCann ,  Govardana Sachithanandam Ramachandran ,  Alexis Roos ,  Caiming Xiong

IPC: G06N3/08 ,  G06F40/40 ,  G06N3/04

Abstract: Embodiments described herein provides a training mechanism that transfers the knowledge from a trained BERT model into a much smaller model to approximate the behavior of BERT. Specifically, the BERT model may be treated as a teacher model, and a much smaller student model may be trained using the same inputs to the teacher model and the output from the teacher model. In this way, the student model can be trained within a much shorter time than the BERT teacher model, but with comparable performance with BERT.

公开(公告)号：US11922303B2

公开(公告)日：2024-03-05

申请号：US16877339

申请日：2020-05-18

Applicant: Salesforce.com, Inc.

Inventor： Wenhao Liu ,  Ka Chun Au ,  Shashank Harinath ,  Bryan McCann ,  Govardana Sachithanandam Ramachandran ,  Alexis Roos ,  Caiming Xiong

IPC: G06N3/00 ,  G06F40/40 ,  G06N3/045 ,  G06N3/08

CPC classification number: G06N3/08 ,  G06F40/40 ,  G06N3/045

Abstract: Embodiments described herein provides a training mechanism that transfers the knowledge from a trained BERT model into a much smaller model to approximate the behavior of BERT. Specifically, the BERT model may be treated as a teacher model, and a much smaller student model may be trained using the same inputs to the teacher model and the output from the teacher model. In this way, the student model can be trained within a much shorter time than the BERT teacher model, but with comparable performance with BERT.

公开(公告)号：US20220245349A1

公开(公告)日：2022-08-04

申请号：US17162318

申请日：2021-01-29

Applicant: salesforce.com, inc.

Inventor： Shiva Kumar Pentyala ,  Jean-Marc Soumet ,  Shashank Harinath ,  Shilpa Bhagavath ,  Johnson Liu ,  Ankit Chadha

IPC: G06F40/30 ,  G06N20/00 ,  G06F40/295

Abstract: Methods and systems for hierarchical natural language understanding are described. A representation of an utterance is inputted to a first machine learning model to obtain information on the first utterance. According to the information on the utterance a determination that the representation of the utterance is to be inputted to a second machine learning model that performs a dedicated natural language task is performed. In response to determining that the representation of the utterance is to be inputted to a second machine learning model, the utterance is inputted to the second machine learning model to obtain an output of the dedicated natural language task.

公开(公告)号：US20230333901A1

公开(公告)日：2023-10-19

申请号：US17659775

申请日：2022-04-19

Applicant: salesforce.com, inc.

Inventor： Arpeet Kale ,  Shashank Harinath

IPC: G06F9/50

CPC classification number: G06F9/5044 ,  G06F9/5055 ,  G06N20/00

Abstract: Techniques are disclosed that pertain to facilitating the execution of machine learning (ML) models. A computer system may implement an ML model layer that permits ML models built using any of a plurality of different ML model frameworks to be submitted without a submitting entity having to define execution logic for a submitted ML model. The computer system may receive, via the ML model layer, configuration metadata for a particular ML model. The computer system may then receive a prediction request from a user to produce a prediction based on the particular ML model. The computer system may produce a prediction based on the particular ML model. As a part of producing that prediction, the computer system may select, in accordance with the received configuration metadata, one of a plurality of types of hardware resources on which to load the particular ML model.

公开(公告)号：US11544465B2

公开(公告)日：2023-01-03

申请号：US17211162

申请日：2021-03-24

Applicant: salesforce.com, inc.

Inventor： Michael Machado ,  John Ball ,  Thomas Archie Cook, Jr. ,  Shashank Harinath ,  Roojuta Lalani ,  Zineb Laraki ,  Qingqing Liu ,  Mike Rosenbaum ,  Karl Ryszard Skucha ,  Jean-Marc Soumet ,  Manju Vijayakumar

IPC: G06F40/295 ,  G06F16/332 ,  G06F40/30 ,  G10L15/18 ,  G10L15/22 ,  G10L15/30

Abstract: Approaches to using unstructured input to update heterogeneous data stores include receiving unstructured text input, receiving a template for interpreting the unstructured text input, identifying, using an entity classifier, entities in the unstructured text input, identifying one or more potential parent entities from the identified entities based on the template, receiving a selection of a parent entity from the one or more potential parent entities, identifying one or more potential child entities from the identified entities based on the template and the selected parent entity, receiving a selection of a child entity from the one or more potential child entities, identifying an action item in the unstructured text input based on the identified entities and the template, determining, using an intent classifier, an intent of the action item, and updating a data store based on the determined intent, the identified entities, and the selected child entity.

公开(公告)号：US11481636B2

公开(公告)日：2022-10-25

申请号：US16877325

申请日：2020-05-18

Applicant: salesforce.com, inc.

Inventor： Govardana Sachithanandam Ramachandran ,  Ka Chun Au ,  Shashank Harinath ,  Wenhao Liu ,  Alexis Roos ,  Caiming Xiong

IPC: G06N3/08 ,  G06N20/00 ,  G06K9/62 ,  G06N20/10 ,  G06F17/18 ,  G06V10/75

Abstract: An embodiment provided herein preprocesses the input samples to the classification neural network, e.g., by adding Gaussian noise to word/sentence representations to make the function of the neural network satisfy Lipschitz property such that a small change in the input does not cause much change to the output if the input sample is in-distribution. Method to induce properties in the feature representation of neural network such that for out-of-distribution examples the feature representation magnitude is either close to zero or the feature representation is orthogonal to all class representations. Method to generate examples that are structurally similar to in-domain and semantically out-of domain for use in out-of-domain classification training. Method to prune feature representation dimension to mitigate long tail error of unused dimension in out-of-domain classification. Using these techniques, the accuracy of both in-domain and out-of-distribution identification can be improved.