公开(公告)号：US12094453B2

公开(公告)日：2024-09-17

申请号：US17447285

申请日：2021-09-09

Applicant: Google LLC

Inventor： Jiahui Yu ,  Chung-cheng Chiu ,  Bo Li ,  Shuo-yiin Chang ,  Tara Sainath ,  Wei Han ,  Anmol Gulati ,  Yanzhang He ,  Arun Narayanan ,  Yonghui Wu ,  Ruoming Pang

IPC: G10L15/06 ,  G10L15/16 ,  G10L15/187 ,  G10L15/22 ,  G10L15/30

CPC classification number: G10L15/063 ,  G10L15/16 ,  G10L15/22 ,  G10L15/30 ,  G10L15/187

Abstract: A computer-implemented method of training a streaming speech recognition model that includes receiving, as input to the streaming speech recognition model, a sequence of acoustic frames. The streaming speech recognition model is configured to learn an alignment probability between the sequence of acoustic frames and an output sequence of vocabulary tokens. The vocabulary tokens include a plurality of label tokens and a blank token. At each output step, the method includes determining a first probability of emitting one of the label tokens and determining a second probability of emitting the blank token. The method also includes generating the alignment probability at a sequence level based on the first probability and the second probability. The method also includes applying a tuning parameter to the alignment probability at the sequence level to maximize the first probability of emitting one of the label tokens.

公开(公告)号：US20240304185A1

公开(公告)日：2024-09-12

申请号：US18598885

申请日：2024-03-07

Applicant: Google LLC

Inventor： Ke Hu ,  Bo Li ,  Tara N. Sainath ,  Yu Zhang ,  Francoise Beaufays

IPC: G10L15/197 ,  G10L15/02 ,  G10L15/06

CPC classification number: G10L15/197 ,  G10L15/02 ,  G10L15/063

Abstract: A method of a multilingual ASR model includes receiving a sequence of acoustic frames characterizing an utterance of speech. At a plurality of output steps, the method further includes generating a first higher order feature representation for an acoustic frame by a first encoder that includes a first plurality of multi-head attention layers; generating a second higher order feature representation for a corresponding first higher order feature representation by a second encoder that includes a second plurality of multi-head attention layers; and generating, by a first decoder, a first probability distribution over possible speech recognition hypotheses based on the second higher order feature representation and a sequence of N previous non-blank symbols. A gating layer of each respective MoE layer configured to dynamically route an output from a previous multi-head attention layer at each of the plurality of output steps to a respective pair of feed-forward expert networks.

公开(公告)号：US20230306958A1

公开(公告)日：2023-09-28

申请号：US18188632

申请日：2023-03-23

Applicant: Google LLC

Inventor： Chao Zhang ,  Bo Li ,  Tara N. Sainath ,  Trevor Strohman ,  Sepand Mavandadi ,  Shuo-yiin Chang ,  Parisa Haghani

IPC: G10L15/00 ,  G10L15/16 ,  G10L15/06

CPC classification number: G10L15/005 ,  G10L15/16 ,  G10L15/063

Abstract: A method includes receiving a sequence of acoustic frames as input to an automatic speech recognition (ASR) model. The method also includes generating, by a first encoder, a first higher order feature representation for a corresponding acoustic frame. The method also includes generating, by a second encoder, a second higher order feature representation for a corresponding first higher order feature representation. The method also includes generating, by a language identification (ID) predictor, a language prediction representation based on a concatenation of the first higher order feature representation and the second higher order feature representation. The method also includes generating, by a first decoder, a first probability distribution over possible speech recognition hypotheses based on a concatenation of the second higher order feature representation and the language prediction representation.

公开(公告)号：US20230237993A1

公开(公告)日：2023-07-27

申请号：US18011571

申请日：2021-10-01

Applicant: Google LLC

Inventor： Jiahui Yu ,  Ruoming Pang ,  Wei Han ,  Anmol Gulati ,  Chung-Cheng Chiu ,  Bo Li ,  Tara N. Sainath ,  Yonghui Hu

IPC: G10L15/16 ,  G10L15/32 ,  G10L15/22

CPC classification number: G10L15/16 ,  G10L15/32 ,  G10L15/22

Abstract: Systems and methods of the present disclosure are directed to a computing system, including one or more processors and a machine-learned multi-mode speech recognition model configured to operate in a streaming recognition mode or a contextual recognition mode. The computing system can perform operations including obtaining speech data and a ground truth label and processing the speech data using the contextual recognition mode to obtain contextual prediction data. The operations can include evaluating a difference between the contextual prediction data and the ground truth label and processing the speech data using the streaming recognition mode to obtain streaming prediction data. The operations can include evaluating a difference between the streaming prediction data and the ground truth label and the contextual and streaming prediction data. The operations can include adjusting parameters of the speech recognition model.

公开(公告)号：US20230147106A1

公开(公告)日：2023-05-11

申请号：US18150724

申请日：2023-01-05

Applicant: GOOGLE LLC

Inventor： Bo Li ,  Kaushik Sheth ,  Edwin Lyle Hudson

IPC: G09G3/36

CPC classification number: G09G3/3688 ,  G09G2360/12

Abstract: A backplane design for delivering image data in an efficient manner to a memory cell forming a part of a pixel driver comprises a word line design and a column data register release signal delivery design that are speed matched and a complementary bit line delivery design that is speed matched to a row decoder signal circuit operative to pull a word line driver to a state to enable the memory circuits of that row to receive data from the column drivers for each column. The speed matching is effective over a range of operating temperatures because the circuit designs are substantially identical.

公开(公告)号：US09984683B2

公开(公告)日：2018-05-29

申请号：US15217457

申请日：2016-07-22

Applicant: Google LLC

Inventor： Bo Li ,  Tara N. Sainath

IPC: G10L15/16 ,  G10L15/02 ,  G06N3/08 ,  G10L15/26

CPC classification number: G10L15/16 ,  G06N3/08 ,  G10L15/02 ,  G10L15/26 ,  G10L2015/025

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for automatic speech recognition using multi-dimensional models. In some implementations, audio data that describes an utterance is received. A transcription for the utterance is determined using an acoustic model that includes a neural network having first memory blocks for time information and second memory blocks for frequency information. The transcription for the utterance is provided as output of an automated speech recognizer.

公开(公告)号：US12073824B2

公开(公告)日：2024-08-27

申请号：US17616135

申请日：2020-12-03

Applicant: GOOGLE LLC

Inventor： Tara N. Sainath ,  Yanzhang He ,  Bo Li ,  Arun Narayanan ,  Ruoming Pang ,  Antoine Jean Bruguier ,  Shuo-Yiin Chang ,  Wei Li

IPC: G10L15/00 ,  G06N3/08 ,  G10L15/05 ,  G10L15/06 ,  G10L15/16 ,  G10L15/22

CPC classification number: G10L15/16 ,  G06N3/08 ,  G10L15/05 ,  G10L15/063 ,  G10L15/22 ,  G10L2015/0635

Abstract: Two-pass automatic speech recognition (ASR) models can be used to perform streaming on-device ASR to generate a text representation of an utterance captured in audio data. Various implementations include a first-pass portion of the ASR model used to generate streaming candidate recognition(s) of an utterance captured in audio data. For example, the first-pass portion can include a recurrent neural network transformer (RNN-T) decoder. Various implementations include a second-pass portion of the ASR model used to revise the streaming candidate recognition(s) of the utterance and generate a text representation of the utterance. For example, the second-pass portion can include a listen attend spell (LAS) decoder. Various implementations include a shared encoder shared between the RNN-T decoder and the LAS decoder.

公开(公告)号：US20240161732A1

公开(公告)日：2024-05-16

申请号：US18418246

申请日：2024-01-20

Applicant: Google LLC

Inventor： Zhifeng Chen ,  Bo Li ,  Eugene Weinstein ,  Yonghui Wu ,  Pedro J. Moreno Mengibar ,  Ron J. Weiss ,  Khe Chai Sim ,  Tara N. Sainath ,  Patrick An Phu Nguyen

IPC: G10L15/00 ,  G10L15/07 ,  G10L15/16

CPC classification number: G10L15/005 ,  G10L15/07 ,  G10L15/16 ,  G10L2015/0631

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer-readable media, for speech recognition using multi-dialect and multilingual models. In some implementations, audio data indicating audio characteristics of an utterance is received. Input features determined based on the audio data are provided to a speech recognition model that has been trained to output score indicating the likelihood of linguistic units for each of multiple different language or dialects. The speech recognition model can be one that has been trained using cluster adaptive training. Output that the speech recognition model generated in response to receiving the input features determined based on the audio data is received. A transcription of the utterance generated based on the output of the speech recognition model is provided.

公开(公告)号：US20240029719A1

公开(公告)日：2024-01-25

申请号：US18340093

申请日：2023-06-23

Applicant: Google LLC

Inventor： Shaan Jagdeep Patrick Bijwadia ,  Shuo-yiin Chang ,  Bo Li ,  Yanzhang He ,  Tara N. Sainath ,  Chao Zhang

IPC: G10L15/16 ,  G10L15/06 ,  G10L25/93

CPC classification number: G10L15/16 ,  G10L15/063 ,  G10L25/93

Abstract: A single E2E multitask model includes a speech recognition model and an endpointer model. The speech recognition model includes an audio encoder configured to encode a sequence of audio frames into corresponding higher-order feature representations, and a decoder configured to generate probability distributions over possible speech recognition hypotheses for the sequence of audio frames based on the higher-order feature representations. The endpointer model is configured to operate between a VAD mode and an EOQ detection mode. During the VAD mode, the endpointer model receives input audio frames, and determines, for each input audio frame, whether the input audio frame includes speech. During the EOQ detection mode, the endpointer model receives latent representations for the sequence of audio frames output from the audio encoder, and determines, for each of the latent representation, whether the latent representation includes final silence.

公开(公告)号：US11676625B2

公开(公告)日：2023-06-13

申请号：US17152918

申请日：2021-01-20

Applicant: Google LLC

Inventor： Shuo-Yiin Chang ,  Bo Li ,  Gabor Simko ,  Maria Carolina Parada San Martin ,  Sean Matthew Shannon

IPC: G10L15/16 ,  G10L25/78 ,  G06N3/08 ,  G06N20/20 ,  G06N5/046 ,  G06F18/214 ,  G06N3/045

CPC classification number: G10L25/78 ,  G06F18/214 ,  G06N3/045 ,  G06N3/08 ,  G06N5/046 ,  G06N20/20 ,  G10L15/16

Abstract: A method for training an endpointer model includes short-form speech utterances and long-form speech utterances. The method also includes providing a short-form speech utterance as input to a shared neural network, the shared neural network configured to learn shared hidden representations suitable for both voice activity detection (VAD) and end-of-query (EOQ) detection. The method also includes generating, using a VAD classifier, a sequence of predicted VAD labels and determining a VAD loss by comparing the sequence of predicted VAD labels to a corresponding sequence of reference VAD labels. The method also includes, generating, using an EOQ classifier, a sequence of predicted EOQ labels and determining an EOQ loss by comparing the sequence of predicted EOQ labels to a corresponding sequence of reference EOQ labels. The method also includes training, using a cross-entropy criterion, the endpointer model based on the VAD loss and the EOQ loss.