公开(公告)号：US11972220B2

公开(公告)日：2024-04-30

申请号：US17456687

申请日：2021-11-29

Applicant: Oracle International Corporation

Inventor： Ying Xu ,  Poorya Zaremoodi ,  Thanh Tien Vu ,  Cong Duy Vu Hoang ,  Vladislav Blinov ,  Yu-Heng Hong ,  Yakupitiyage Don Thanuja Samodhye Dharmasiri ,  Vishal Vishnoi ,  Elias Luqman Jalaluddin ,  Manish Parekh ,  Thanh Long Duong ,  Mark Edward Johnson

IPC: G06F40/35 ,  G06F40/205 ,  G06F40/253 ,  G06N3/08 ,  H04L51/02

CPC classification number: G06F40/35 ,  G06N3/08 ,  H04L51/02 ,  G06F40/205 ,  G06F40/253

Abstract: Techniques for using enhanced logit values for classifying utterances and messages input to chatbot systems in natural language processing. A method can include a chatbot system receiving an utterance generated by a user interacting with the chatbot system and inputting the utterance into a machine-learning model including a series of network layers. A final network layer of the series of network layers can include a logit function. The machine-learning model can map a first probability for a resolvable class to a first logit value using the logit function. The machine-learning model can map a second probability for a unresolvable class to an enhanced logit value. The method can also include the chatbot system classifying the utterance as the resolvable class or the unresolvable class based on the first logit value and the enhanced logit value.

公开(公告)号：US20210304733A1

公开(公告)日：2021-09-30

申请号：US17016117

申请日：2020-09-09

Applicant: Oracle International Corporation

Inventor： Elias Luqman Jalaluddin ,  Vishal Vishnoi ,  Mark Edward Johnson ,  Thanh Long Duong ,  Yu-Heng Hong ,  Balakota Srinivas Vinnakota

IPC: G10L15/06 ,  G10L15/22 ,  G10L15/26 ,  G10L15/05 ,  G10L15/18

Abstract: Techniques for noise data augmentation for training chatbot systems in natural language processing. In one particular aspect, a method is provided that includes receiving a training set of utterances for training an intent classifier to identify one or more intents for one or more utterances; augmenting the training set of utterances with noise text to generate an augmented training set of utterances; and training the intent classifier using the augmented training set of utterances. The augmenting includes: obtaining the noise text from a list of words, a text corpus, a publication, a dictionary, or any combination thereof irrelevant of original text within the utterances of the training set of utterances, and incorporating the noise text within the utterances relative to the original text in the utterances of the training set of utterances at a predefined augmentation ratio to generate augmented utterances.

公开(公告)号：US20210304074A1

公开(公告)日：2021-09-30

申请号：US17216498

申请日：2021-03-29

Applicant: Oracle International Corporation

Inventor： Poorya Zaremoodi ,  Ying Xu ,  Thanh Tien Vu ,  Vladislav Blinov ,  Yu-Heng Hong ,  Yakupitiyage Don Thanuja Samodhye Dharmasiri ,  Vishal Vishnoi ,  Elias Luqman Jalaluddin ,  Manish Parekh ,  Thanh Long Duong ,  Mark Edward Johnson ,  Xin Xu ,  Cong Duy Vu Hoang

IPC: G06N20/00

Abstract: Techniques are disclosed for tuning hyperparameters of a machine-learning model. A plurality of metrics are selected for which hyperparameters of the machine-learning model are to be tuned. Each metric is associated with a plurality of specification parameters including a target score, a penalty factor, and a bonus factor. The plurality of specification parameters are configured for each metric in accordance with a first criterion. The machine-learning model is evaluated using one or more validation datasets to obtain a metric score. A weighted loss function is formulated based on a difference between the metric score and the target score of each metric, the penalty factor or the bonus factor. The hyperparameters associated with the machine-learning model are tuned in order to optimize the weighted loss function. In response to the weighted loss function being optimized, the machine-learning model is provided as a validated machine-learning model.

公开(公告)号：US12293155B2

公开(公告)日：2025-05-06

申请号：US18630772

申请日：2024-04-09

Applicant: Oracle International Corporation

Inventor： Elias Luqman Jalaluddin ,  Vishal Vishnoi ,  Thanh Long Duong ,  Mark Edward Johnson ,  Poorya Zaremoodi ,  Gautam Singaraju ,  Ying Xu ,  Vladislav Blinov ,  Yu-Heng Hong

IPC: G06F40/289 ,  G06F40/30 ,  G06N3/08 ,  H04L51/02

Abstract: A method includes receiving a training set of utterances for training a machine-learning model to identify one or more intents for one or more utterances, and augmenting the training set of utterances with out-of-domain (OOD) examples. The augmenting includes: generating a data set of OOD examples, filtering out OOD examples from the data set of OOD examples, determining a difficulty value for each OOD example remaining within the filtered data set of the OOD examples, and generating augmented batches of utterances including utterances from the training set of utterances and utterances from the filtered data set of the OOD based on the difficulty value for each OOD. Thereafter, the machine-learning model is trained using the augmented batches of utterances in accordance with a curriculum training protocol.

公开(公告)号：US20240256777A1

公开(公告)日：2024-08-01

申请号：US18630772

申请日：2024-04-09

Applicant: Oracle International Corporation

Inventor： Elias Luqman Jalaluddin ,  Vishal Vishnoi ,  Thanh Long Duong ,  Mark Edward Johnson ,  Poorya Zaremoodi ,  Gautam Singaraju ,  Ying Xu ,  Vladislav Blinov ,  Yu-Heng Hong

IPC: G06F40/289 ,  G06F40/30 ,  G06N3/08 ,  H04L51/02

CPC classification number: G06F40/289 ,  G06F40/30 ,  G06N3/08 ,  H04L51/02

Abstract: A method includes receiving a training set of utterances for training a machine-learning model to identify one or more intents for one or more utterances, and augmenting the training set of utterances with out-of-domain (OOD) examples. The augmenting includes: generating a data set of OOD examples, filtering out OOD examples from the data set of OOD examples, determining a difficulty value for each OOD example remaining within the filtered data set of the OOD examples, and generating augmented batches of utterances including utterances from the training set of utterances and utterances from the filtered data set of the OOD based on the difficulty value for each OOD. Thereafter, the machine-learning model is trained using the augmented batches of utterances in accordance with a curriculum training protocol.

公开(公告)号：US20240232541A1

公开(公告)日：2024-07-11

申请号：US18611039

申请日：2024-03-20

Applicant: Oracle International Corporation

Inventor： Ying Xu ,  Poorya Zaremoodi ,  Thanh Tien Vu ,  Cong Duy Vu Hoang ,  Vladislav Blinov ,  Yu-Heng Hong ,  Yakupitiyage Don Thanuja Samodhye Dharmasiri ,  Vishal Vishnoi ,  Elias Luqman Jalaluddin ,  Manish Parekh ,  Thanh Long Duong ,  Mark Edward Johnson

IPC: G06F40/35 ,  G06F40/205 ,  G06F40/253 ,  G06N3/08 ,  H04L51/02

CPC classification number: G06F40/35 ,  G06N3/08 ,  H04L51/02 ,  G06F40/205 ,  G06F40/253

Abstract: Techniques for using enhanced logit values for classifying utterances and messages input to chatbot systems in natural language processing. A method can include a chatbot system receiving an utterance generated by a user interacting with the chatbot system and inputting the utterance into a machine-learning model including a series of network layers. A final network layer of the series of network layers can include a logit function. The machine-learning model can map a first probability for a resolvable class to a first logit value using the logit function. The machine-learning model can map a second probability for a unresolvable class to an enhanced logit value. The method can also include the chatbot system classifying the utterance as the resolvable class or the unresolvable class based on the first logit value and the enhanced logit value.

公开(公告)号：US11972755B2

公开(公告)日：2024-04-30

申请号：US17993130

申请日：2022-11-23

Applicant: Oracle International Corporation

Inventor： Elias Luqman Jalaluddin ,  Vishal Vishnoi ,  Mark Edward Johnson ,  Thanh Long Duong ,  Yu-Heng Hong ,  Balakota Srinivas Vinnakota

IPC: G10L15/22 ,  G10L15/05 ,  G10L15/06 ,  G10L15/18 ,  G10L15/26

CPC classification number: G10L15/063 ,  G10L15/05 ,  G10L15/18 ,  G10L15/22 ,  G10L15/26 ,  G10L2015/0633 ,  G10L2015/0638 ,  G10L2015/227

Abstract: Techniques for noise data augmentation for training chatbot systems in natural language processing. In one particular aspect, a method is provided that includes receiving a training set of utterances for training an intent classifier to identify one or more intents for one or more utterances; augmenting the training set of utterances with noise text to generate an augmented training set of utterances; and training the intent classifier using the augmented training set of utterances. The augmenting includes: obtaining the noise text from a list of words, a text corpus, a publication, a dictionary, or any combination thereof irrelevant of original text within the utterances of the training set of utterances, and incorporating the noise text within the utterances relative to the original text in the utterances of the training set of utterances at a predefined augmentation ratio to generate augmented utterances.

公开(公告)号：US20220172021A1

公开(公告)日：2022-06-02

申请号：US17455181

申请日：2021-11-16

Applicant: Oracle International Corporation

Inventor： Cong Duy Vu Hoang ,  Thanh Tien Vu ,  Poorya Zaremoodi ,  Ying Xu ,  Vladislav Blinov ,  Yu-Heng Hong ,  Yakupitiyage Don Thanuja Samodhye Dharmasiri ,  Vishal Vishnoi ,  Elias Luqman Jalaluddin ,  Manish Parekh ,  Thanh Long Duong ,  Mark Edward Johnson

IPC: G06N3/04 ,  G06N3/08

Abstract: Disclosed herein are techniques for addressing an overconfidence problem associated with machine learning models in chatbot systems. For each layer of a plurality of layers of a machine learning model, a distribution of confidence scores is generated for a plurality of predictions with respect to an input utterance. A prediction is determined for each layer of the machine learning model based on the distribution of confidence scores generated for the layer. Based on the predictions, an overall prediction of the machine learning model is determined. A subset of the plurality of layers are iteratively processed to identify a layer whose assigned prediction satisfies a criterion. A confidence score associated with the assigned prediction of the layer of the machine learning model is assigned as an overall confidence score to be associated with the overall prediction of the machine learning model.

公开(公告)号：US20220171946A1

公开(公告)日：2022-06-02

申请号：US17456687

申请日：2021-11-29

Applicant: Oracle International Corporation

Inventor： Ying Xu ,  Poorya Zaremoodi ,  Thanh Tien Vu ,  Cong Duy Vu Hoang ,  Vladislav Blinov ,  Yu-Heng Hong ,  Yakupitiyage Don Thanuja Samodhye Dharmasiri ,  Vishal Vishnoi ,  Elias Luqman Jalaluddin ,  Manish Parekh ,  Thanh Long Duong ,  Mark Edward Johnson

IPC: G06F40/35 ,  H04L51/02 ,  G06N3/08

Abstract: Techniques for using enhanced logit values for classifying utterances and messages input to chatbot systems in natural language processing. A method can include a chatbot system receiving an utterance generated by a user interacting with the chatbot system and inputting the utterance into a machine-learning model including a series of network layers. A final network layer of the series of network layers can include a logit function. The machine-learning model can map a first probability for a resolvable class to a first logit value using the logit function. The machine-learning model can map a second probability for a unresolvable class to an enhanced logit value. The method can also include the chatbot system classifying the utterance as the resolvable class or the unresolvable class based on the first logit value and the enhanced logit value.

公开(公告)号：US11790901B2

公开(公告)日：2023-10-17

申请号：US18092170

申请日：2022-12-30

Applicant: Oracle International Corporation

Inventor： Thanh Long Duong ,  Mark Edward Johnson ,  Vu Cong Duy Hoang ,  Tuyen Quang Pham ,  Yu-Heng Hong ,  Vladislavs Dovgalecs ,  Guy Bashkansky ,  Jason Eric Black ,  Andrew David Bleeker ,  Serge Le Huitouze

IPC: G10L15/22

CPC classification number: G10L15/22

Abstract: Described herein are dialog systems, and techniques for providing such dialog systems, that are suitable for use on standalone computing devices. In some embodiments, a dialog system includes a dialog manager, which takes as input an input logical form, which may be a representation of user input. The dialog manager may include a dialog state tracker, an execution subsystem, a dialog policy subsystem, and a context stack. The dialog state tracker may generate an intermediate logical form from the input logical form combined with a context from the context stack. The context stack may maintain a history of a current dialog, and thus, the intermediate logical form may include contextual information potentially missing from the input logical form. The execution subsystem may execute the intermediate logical form to produce an execution result, and the dialog policy subsystem may generate an output logical form based on the execution result.