公开(公告)号：US10181098B2

公开(公告)日：2019-01-15

申请号：US14731326

申请日：2015-06-04

Applicant: GOOGLE LLC

Inventor： Oriol Vinyals ,  Quoc V. Le ,  Ilya Sutskever

IPC: G06N3/02 ,  G06N3/04 ,  G06F17/28

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating representations of input sequences. One of the methods includes obtaining an input sequence, the input sequence comprising a plurality of inputs arranged according to an input order; processing the input sequence using a first long short term memory (LSTM) neural network to convert the input sequence into an alternative representation for the input sequence; and processing the alternative representation for the input sequence using a second LSTM neural network to generate a target sequence for the input sequence, the target sequence comprising a plurality of outputs arranged according to an output order.

公开(公告)号：US10083169B1

公开(公告)日：2018-09-25

申请号：US15248966

申请日：2016-08-26

Applicant: Google LLC

Inventor： Shalini Ghosh ,  Oriol Vinyals ,  Brian Patrick Strope ,  Howard Scott Roy ,  Thomas L. Dean ,  Larry Paul Heck

IPC: G10L15/00 ,  G06F17/27 ,  G06N3/08 ,  G06N3/04

CPC classification number: G06F17/279 ,  G06F17/2881 ,  G06N3/0445 ,  G06N3/084

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing word sequences using neural networks. One of the methods includes receiving a first sequence of words arranged according to a first order; and for each word in the first sequence, beginning with a first word in the first order: determining a topic vector that is associated with the word; generating a combined input from the word and the topic vector, and processing the combined input through one or more sequence modeling layers to generate a sequence modeling output for the word; and processing one or more of the sequence modeling outputs through an output layer to generate a neural network output for the first sequence of words.

公开(公告)号：US12014259B2

公开(公告)日：2024-06-18

申请号：US17092837

申请日：2020-11-09

Applicant: Google LLC

Inventor： Samy Bengio ,  Oriol Vinyals ,  Alexander Toshkov Toshev ,  Dumitru Erhan

IPC: G06N3/00 ,  G06F40/40 ,  G06N3/045 ,  G06N3/047

CPC classification number: G06N3/047 ,  G06F40/40 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating descriptions of input images. One of the methods includes obtaining an input image; processing the input image using a first neural network to generate an alternative representation for the input image; and processing the alternative representation for the input image using a second neural network to generate a sequence of a plurality of words in a target natural language that describes the input image.

公开(公告)号：US11860969B2

公开(公告)日：2024-01-02

申请号：US16989455

申请日：2020-08-10

Applicant: Google LLC

Inventor： Mostafa Dehghani ,  Stephan Gouws ,  Oriol Vinyals ,  Jakob D. Uszkoreit ,  Lukasz Mieczyslaw Kaiser

IPC: G06N3/04 ,  G06F17/14

CPC classification number: G06F17/14 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for implementing a sequence to sequence model that is recurrent in depth while employing self-attention to combine information from different parts of sequences.

公开(公告)号：US11829860B2

公开(公告)日：2023-11-28

申请号：US17679625

申请日：2022-02-24

Applicant: Google LLC

Inventor： Oriol Vinyals ,  Samuel Bengio

IPC: G06N3/04 ,  G06N3/044 ,  G06N3/045

CPC classification number: G06N3/044 ,  G06N3/045

Abstract: In one aspect, this specification describes a recurrent neural network system implemented by one or more computers that is configured to process input sets to generate neural network outputs for each input set. The input set can be a collection of multiple inputs for which the recurrent neural network should generate the same neural network output regardless of the order in which the inputs are arranged in the collection. The recurrent neural network system can include a read neural network, a process neural network, and a write neural network. In another aspect, this specification describes a system implemented as computer programs on one or more computers in one or more locations that is configured to train a recurrent neural network that receives a neural network input and sequentially emits outputs to generate an output sequence for the neural network input.

公开(公告)号：US20220351091A1

公开(公告)日：2022-11-03

申请号：US17863733

申请日：2022-07-13

Applicant: Google LLC

Inventor： Oriol Vinyals ,  Jeffrey Adgate Dean ,  Geoffrey E. Hinton

IPC: G06N20/20 ,  G06N7/00 ,  G06N20/00 ,  G06N3/04 ,  G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a distilled machine learning model. One of the methods includes training a cumbersome machine learning model, wherein the cumbersome machine learning model is configured to receive an input and generate a respective score for each of a plurality of classes; and training a distilled machine learning model on a plurality of training inputs, wherein the distilled machine learning model is also configured to receive inputs and generate scores for the plurality of classes, comprising: processing each training input using the cumbersome machine learning model to generate a cumbersome target soft output for the training input; and training the distilled machine learning model to, for each of the training inputs, generate a soft output that matches the cumbersome target soft output for the training input.

公开(公告)号：US20220028375A1

公开(公告)日：2022-01-27

申请号：US17450235

申请日：2021-10-07

Applicant: Google LLC

Inventor： William Chan ,  Navdeep Jaitly ,  Quoc V. Le ,  Oriol Vinyals ,  Noam M. Shazeer

IPC: G10L15/16 ,  G06N3/04 ,  G06F40/12 ,  G06F40/197 ,  G10L15/183 ,  G10L15/26

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media for speech recognition. One method includes obtaining an input acoustic sequence, the input acoustic sequence representing an utterance, and the input acoustic sequence comprising a respective acoustic feature representation at each of a first number of time steps; processing the input acoustic sequence using a first neural network to convert the input acoustic sequence into an alternative representation for the input acoustic sequence; processing the alternative representation for the input acoustic sequence using an attention-based Recurrent Neural Network (RNN) to generate, for each position in an output sequence order, a set of substring scores that includes a respective substring score for each substring in a set of substrings; and generating a sequence of substrings that represent a transcription of the utterance.

公开(公告)号：US11222252B2

公开(公告)日：2022-01-11

申请号：US16211635

申请日：2018-12-06

Applicant: Google LLC

Inventor： Oriol Vinyals ,  Quoc V. Le ,  Ilya Sutskever

IPC: G06N3/02 ,  G06N3/04 ,  G06F40/40

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating representations of input sequences. One of the methods includes obtaining an input sequence, the input sequence comprising a plurality of inputs arranged according to an input order; processing the input sequence using a first long short term memory (LSTM) neural network to convert the input sequence into an alternative representation for the input sequence; and processing the alternative representation for the input sequence using a second LSTM neural network to generate a target sequence for the input sequence, the target sequence comprising a plurality of outputs arranged according to an output order.

公开(公告)号：US10936828B2

公开(公告)日：2021-03-02

申请号：US16193387

申请日：2018-11-16

Applicant: Google LLC

Inventor： Quoc V. Le ,  Minh-Thang Luong ,  Ilya Sutskever ,  Oriol Vinyals ,  Wojciech Zaremba

IPC: G06F17/28 ,  G06F40/56 ,  G06N3/04 ,  G06F40/44 ,  G06F40/45 ,  G06F40/242 ,  G06F7/02 ,  G06F7/10 ,  G10L15/02 ,  G10L15/16

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for neural translation systems with rare word processing. One of the methods is a method training a neural network translation system to track the source in source sentences of unknown words in target sentences, in a source language and a target language, respectively and includes deriving alignment data from a parallel corpus, the alignment data identifying, in each pair of source and target language sentences in the parallel corpus, aligned source and target words; annotating the sentences in the parallel corpus according to the alignment data and a rare word model to generate a training dataset of paired source and target language sentences; and training a neural network translation model on the training dataset.

公开(公告)号：US20200042866A1

公开(公告)日：2020-02-06

申请号：US16538712

申请日：2019-08-12

Applicant: Google LLC

Inventor： Samuel Bengio ,  Oriol Vinyals ,  Alexander Toshkov Toshev ,  Dumitru Erhan

IPC: G06N3/04 ,  G06F17/28

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating descriptions of input images. One of the methods includes obtaining an input image; processing the input image using a first neural network to generate an alternative representation for the input image; and processing the alternative representation for the input image using a second neural network to generate a sequence of a plurality of words in a target natural language that describes the input image.