公开(公告)号：US12106749B2

公开(公告)日：2024-10-01

申请号：US17448119

申请日：2021-09-20

Applicant: Google LLC

Inventor： Rohit Prakash Prabhavalkar ,  Zhifeng Chen ,  Bo Li ,  Chung-cheng Chiu ,  Kanury Kanishka Rao ,  Yonghui Wu ,  Ron J. Weiss ,  Navdeep Jaitly ,  Michiel A. u. Bacchiani ,  Tara N. Sainath ,  Jan Kazimierz Chorowski ,  Anjuli Patricia Kannan ,  Ekaterina Gonina ,  Patrick An Phu Nguyen

IPC: G10L15/00 ,  G06N3/08 ,  G10L15/02 ,  G10L15/06 ,  G10L15/16 ,  G10L15/22 ,  G10L25/30 ,  G10L15/26

CPC classification number: G10L15/16 ,  G06N3/08 ,  G10L15/02 ,  G10L15/063 ,  G10L15/22 ,  G10L25/30 ,  G10L2015/025 ,  G10L15/26

Abstract: A method for performing speech recognition using sequence-to-sequence models includes receiving audio data for an utterance and providing features indicative of acoustic characteristics of the utterance as input to an encoder. The method also includes processing an output of the encoder using an attender to generate a context vector, generating speech recognition scores using the context vector and a decoder trained using a training process, and generating a transcription for the utterance using word elements selected based on the speech recognition scores. The transcription is provided as an output of the ASR system.

公开(公告)号：US11908461B2

公开(公告)日：2024-02-20

申请号：US17149018

申请日：2021-01-14

Applicant: Google LLC

Inventor： Ke Hu ,  Tara N. Sainath ,  Ruoming Pang ,  Rohit Prakash Prabhavalkar

IPC: G10L15/18 ,  G06N3/049 ,  G10L15/06 ,  G10L15/16 ,  G10L15/187 ,  G10L19/00

CPC classification number: G10L15/1815 ,  G06N3/049 ,  G10L15/063 ,  G10L15/16 ,  G10L15/187 ,  G10L19/0018

Abstract: A method of performing speech recognition using a two-pass deliberation architecture includes receiving a first-pass hypothesis and an encoded acoustic frame and encoding the first-pass hypothesis at a hypothesis encoder. The first-pass hypothesis is generated by a recurrent neural network (RNN) decoder model for the encoded acoustic frame. The method also includes generating, using a first attention mechanism attending to the encoded acoustic frame, a first context vector, and generating, using a second attention mechanism attending to the encoded first-pass hypothesis, a second context vector. The method also includes decoding the first context vector and the second context vector at a context vector decoder to form a second-pass hypothesis.

公开(公告)号：US20230343332A1

公开(公告)日：2023-10-26

申请号：US18304064

申请日：2023-04-20

Applicant: Google LLC

Inventor： Ronny Huang ,  Shuo-yiin Chang ,  David Rybach ,  Rohit Prakash Prabhavalkar ,  Tara N. Sainath ,  Cyril Allauzen ,  Charles Caleb Peyser ,  Zhiyun Lu

IPC: G10L15/04 ,  G10L25/93 ,  G10L15/197 ,  G10L15/06 ,  G10L15/22 ,  G10L15/02

CPC classification number: G10L15/197 ,  G10L15/02 ,  G10L15/04 ,  G10L15/063 ,  G10L15/22 ,  G10L25/93 ,  G10L2015/025 ,  G10L2025/932

Abstract: A joint segmenting and ASR model includes an encoder and decoder. The encoder configured to: receive a sequence of acoustic frames characterizing one or more utterances; and generate, at each output step, a higher order feature representation for a corresponding acoustic frame. The decoder configured to: receive the higher order feature representation and generate, at each output step: a probability distribution over possible speech recognition hypotheses, and an indication of whether the corresponding output step corresponds to an end of speech segment. The j oint segmenting and ASR model trained on a set of training samples, each training sample including: audio data characterizing a spoken utterance; and a corresponding transcription of the spoken utterance, the corresponding transcription having an end of speech segment ground truth token inserted into the corresponding transcription automatically based on a set of heuristic-based rules and exceptions applied to the training sample.

公开(公告)号：US20220172706A1

公开(公告)日：2022-06-02

申请号：US17651315

申请日：2022-02-16

Applicant: Google LLC

Inventor： Ke Hu ,  Golan Pundak ,  Rohit Prakash Prabhavalkar ,  Antoine Jean Bruguier ,  Tara N. Sainath

IPC: G10L15/06 ,  G10L15/02 ,  G10L15/187 ,  G10L15/193 ,  G10L15/28 ,  G10L15/32 ,  G10L25/30

Abstract: A method includes receiving audio data encoding an utterance spoken by a native speaker of a first language, and receiving a biasing term list including one or more terms in a second language different than the first language. The method also includes processing, using a speech recognition model, acoustic features derived from the audio data to generate speech recognition scores for both wordpieces and corresponding phoneme sequences in the first language. The method also includes rescoring the speech recognition scores for the phoneme sequences based on the one or more terms in the biasing term list, and executing, using the speech recognition scores for the wordpieces and the rescored speech recognition scores for the phoneme sequences, a decoding graph to generate a transcription for the utterance.

公开(公告)号：US11145293B2

公开(公告)日：2021-10-12

申请号：US16516390

申请日：2019-07-19

Applicant: Google LLC

Inventor： Rohit Prakash Prabhavalkar ,  Zhifeng Chen ,  Bo Li ,  Chung-Cheng Chiu ,  Kanury Kanishka Rao ,  Yonghui Wu ,  Ron J. Weiss ,  Navdeep Jaitly ,  Michiel A. U. Bacchiani ,  Tara N. Sainath ,  Jan Kazimierz Chorowski ,  Anjuli Patricia Kannan ,  Ekaterina Gonina ,  Patrick An Phu Nguyen

IPC: G10L15/00 ,  G10L15/16 ,  G10L15/22 ,  G10L15/02 ,  G06N3/08 ,  G10L15/06 ,  G10L25/30 ,  G10L15/26

Abstract: Methods, systems, and apparatus, including computer-readable media, for performing speech recognition using sequence-to-sequence models. An automated speech recognition (ASR) system receives audio data for an utterance and provides features indicative of acoustic characteristics of the utterance as input to an encoder. The system processes an output of the encoder using an attender to generate a context vector and generates speech recognition scores using the context vector and a decoder trained using a training process that selects at least one input to the decoder with a predetermined probability. An input to the decoder during training is selected between input data based on a known value for an element in a training example, and input data based on an output of the decoder for the element in the training example. A transcription is generated for the utterance using word elements selected based on the speech recognition scores. The transcription is provided as an output of the ASR system.

公开(公告)号：US20210225369A1

公开(公告)日：2021-07-22

申请号：US17149018

申请日：2021-01-14

Applicant: Google LLC

Inventor： Ke Hu ,  Tara N. Sainath ,  Ruoming Pang ,  Rohit Prakash Prabhavalkar

IPC: G10L15/18 ,  G10L15/06 ,  G06N3/04 ,  G10L15/187 ,  G10L15/16 ,  G10L19/00

Abstract: A method of performing speech recognition using a two-pass deliberation architecture includes receiving a first-pass hypothesis and an encoded acoustic frame and encoding the first-pass hypothesis at a hypothesis encoder. The first-pass hypothesis is generated by a recurrent neural network (RNN) decoder model for the encoded acoustic frame. The method also includes generating, using a first attention mechanism attending to the encoded acoustic frame, a first context vector, and generating, using a second attention mechanism attending to the encoded first-pass hypothesis, a second context vector. The method also includes decoding the first context vector and the second context vector at a context vector decoder to form a second-pass hypothesis

公开(公告)号：US20210090570A1

公开(公告)日：2021-03-25

申请号：US16580726

申请日：2019-09-24

Applicant: Google LLC

Inventor： Asaf Aharoni ,  Arun Narayanan ,  Nir Shabat ,  Parisa Haghani ,  Galen Tsai Chuang ,  Yaniv Leviathan ,  Neeraj Gaur ,  Pedro J. Moreno Mengibar ,  Rohit Prakash Prabhavalkar ,  Zhongdi Qu ,  Austin Severn Waters ,  Tomer Amiaz ,  Michiel A.U. Bacchiani

IPC: G10L15/26 ,  H04M3/428 ,  H04M1/02 ,  G10L15/32 ,  H04M3/51 ,  H04M1/663

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for an automated calling system are disclosed. In one aspect, a method includes the actions of receiving audio data of an utterance spoken by a user who is having a telephone conversation with a bot. The actions further include determining a context of the telephone conversation. The actions further include determining a user intent of a first previous portion of the telephone conversation spoken by the user and a bot intent of a second previous portion of the telephone conversation outputted by a speech synthesizer of the bot. The actions further include, based on the audio data of the utterance, the context of the telephone conversation, the user intent, and the bot intent, generating synthesized speech of a reply by the bot to the utterance. The actions further include, providing, for output, the synthesized speech.

公开(公告)号：US20240221750A1

公开(公告)日：2024-07-04

申请号：US18610233

申请日：2024-03-19

Applicant: Google LLC

Inventor： Wei Li ,  Rohit Prakash Prabhavalkar ,  Kanury Kanishka Rao ,  Yanzhang He ,  Ian C. McGraw ,  Anton Bakhtin

IPC: G10L15/22 ,  G10L15/02 ,  G10L15/06 ,  G10L15/08 ,  G10L15/14 ,  G10L15/18 ,  G10L19/00

CPC classification number: G10L15/22 ,  G10L15/02 ,  G10L15/063 ,  G10L15/18 ,  G10L19/00 ,  G10L2015/025 ,  G10L2015/088 ,  G10L15/142 ,  G10L2015/223

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for detecting utterances of a key phrase in an audio signal. One of the methods includes receiving, by a key phrase spotting system, an audio signal encoding one or more utterances; while continuing to receive the audio signal, generating, by the key phrase spotting system, an attention output using an attention mechanism that is configured to compute the attention output based on a series of encodings generated by an encoder comprising one or more neural network layers; generating, by the key phrase spotting system and using attention output, output that indicates whether the audio signal likely encodes the key phrase; and providing, by the key phrase spotting system, the output that indicates whether the audio signal likely encodes the key phrase.

公开(公告)号：US11948062B2

公开(公告)日：2024-04-02

申请号：US17112966

申请日：2020-12-04

Applicant: Google LLC

Inventor： Ouais Alsharif ,  Rohit Prakash Prabhavalkar ,  Ian C. McGraw ,  Antoine Jean Bruguier

IPC: G06N3/044 ,  G06N3/049 ,  G06N3/08 ,  G06N20/00 ,  G06F17/16 ,  G06N3/04 ,  G06N3/084

CPC classification number: G06N3/044 ,  G06N3/049 ,  G06N3/08 ,  G06N20/00 ,  G05B2219/33025 ,  G05B2219/40326 ,  G06F17/16 ,  G06N3/04 ,  G06N3/084

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for implementing a compressed recurrent neural network (RNN). One of the systems includes a compressed RNN, the compressed RNN comprising a plurality of recurrent layers, wherein each of the recurrent layers has a respective recurrent weight matrix and a respective inter-layer weight matrix, and wherein at least one of recurrent layers is compressed such that a respective recurrent weight matrix of the compressed layer is defined by a first compressed weight matrix and a projection matrix and a respective inter-layer weight matrix of the compressed layer is defined by a second compressed weight matrix and the projection matrix.

公开(公告)号：US11942076B2

公开(公告)日：2024-03-26

申请号：US17651315

申请日：2022-02-16

Applicant: Google LLC

Inventor： Ke Hu ,  Golan Pundak ,  Rohit Prakash Prabhavalkar ,  Antoine Jean Bruguier ,  Tara N. Sainath

IPC: G10L15/30 ,  G10L15/02 ,  G10L15/06 ,  G10L15/187 ,  G10L15/193 ,  G10L15/28 ,  G10L15/32 ,  G10L25/30

CPC classification number: G10L15/063 ,  G10L15/02 ,  G10L15/187 ,  G10L15/193 ,  G10L15/285 ,  G10L15/32 ,  G10L25/30 ,  G10L2015/025

Abstract: A method includes receiving audio data encoding an utterance spoken by a native speaker of a first language, and receiving a biasing term list including one or more terms in a second language different than the first language. The method also includes processing, using a speech recognition model, acoustic features derived from the audio data to generate speech recognition scores for both wordpieces and corresponding phoneme sequences in the first language. The method also includes rescoring the speech recognition scores for the phoneme sequences based on the one or more terms in the biasing term list, and executing, using the speech recognition scores for the wordpieces and the rescored speech recognition scores for the phoneme sequences, a decoding graph to generate a transcription for the utterance.