公开(公告)号：US20240330762A1

公开(公告)日：2024-10-03

申请号：US18293638

申请日：2021-09-03

Applicant: Google LLC

Inventor： Dana Alon ,  Yair Alon ,  Elad Eban

IPC: G06N20/00

CPC classification number: G06N20/00

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for improving the representation of items of a vocabulary in an embedding space for use in machine learning models. An embedding matrix is generated wherein each row in the embedding matrix is a vector of elements and corresponds to an item of a vocabulary. A score is assigned to each vector in the embedding matrix indicating a probability of its corresponding vector being used in the machine learning model. The scores are iteratively updated by sampling a proper subset of vectors and updating the elements of each respective vector in the proper subset of vectors based on the respective scores of vectors. The score of each vector are then updated based on a loss function of the machine learning model. The embedding matrix is then re-structured based on the updated scores of the vectors.

公开(公告)号：US20220156524A1

公开(公告)日：2022-05-19

申请号：US17526886

申请日：2021-11-15

Applicant: Google LLC

Inventor： Yair Alon ,  Elad Eban ,  Xiaofeng Wang

IPC: G06K9/62 ,  G06N3/04

Abstract: A combination of two or more trained machine learning models can exhibit a combined accuracy greater than the accuracy of any one of the constituent models. However, this increase accuracy comes at additional computational cost. Cascades of machine learning models are provided herein that result in increased model accuracy and/or reduced model compute cost. These benefits are obtained by conditionally executing one or more of the models of the cascade based on the estimated correctness of already-executed models. The estimated correctness can be obtained as an additional output of the already-executed model(s) or could be determined as an entropy, maximum class probability, maximum class logit, or other function of the output(s) of the already-executed model(s). The expected computational cost of executing the model cascade is reduced by only executing the downstream model(s) when the upstream model(s) has resulted in an output whose accuracy is suspect.

公开(公告)号：US11928601B2

公开(公告)日：2024-03-12

申请号：US15892890

申请日：2018-02-09

Applicant: Google LLC

Inventor： Yair Alon ,  Elad Eban

IPC: G06N3/084 ,  G06N3/044 ,  G06N3/08 ,  G06N7/01 ,  G06N20/00 ,  G06N3/063

CPC classification number: G06N3/084 ,  G06N3/044 ,  G06N3/08 ,  G06N7/01 ,  G06N3/063 ,  G06N20/00

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for neural network compression. In one aspect, a method comprises receiving a neural network and identifying a particular set of multiple weights of the neural network. Multiple anchor points are determined based on current values of the particular set of weights of the neural network. The neural network is trained by, at each of multiple training iterations, performing operations comprising adjusting the values of the particular set of weights by backpropagating gradients of a loss function. The loss function comprises a first loss function term based on a prediction accuracy of the neural network and a second loss function term based on a similarity of the current values of the particular set of weights to the anchor points. After training, the values of the particular set of weights are quantized based on the anchor points.

公开(公告)号：US20230063686A1

公开(公告)日：2023-03-02

申请号：US17797996

申请日：2021-02-08

Applicant: Google LLC

Inventor： Hanhan Li ,  Max Moroz ,  Shraman Ray Chaudhuri ,  Yair Alon ,  Elad Eban

IPC: G06N3/08 ,  G06K9/62

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining neural network architectures. One of the methods includes receiving training data; receiving architecture data; assigning, to each of a plurality of network operators, a utilization variable indicating a likelihood of the network operator being utilized in a neural network; generating an optimized neural network for performing the neural network task, comprising, repeatedly performing the following: sampling a selected set of network operators; and training the neural network having an architecture defined by the selected set of network operators, wherein the training comprises: computing an objective function evaluating (i) a measure of computational cost of the neural network and (ii) a measure of performance of the neural network on the neural network task associated with the training data; and adjusting the respective current values of the utilization variables and respective current values of the neural network parameters.

公开(公告)号：US11875262B2

公开(公告)日：2024-01-16

申请号：US17701778

申请日：2022-03-23

Applicant: Google LLC

Inventor： Ofir Nachum ,  Ariel Gordon ,  Elad Eban ,  Bo Chen

IPC: G06N3/082 ,  G06N3/084 ,  G06N3/045 ,  G06N3/047 ,  G06N20/00

CPC classification number: G06N3/082 ,  G06N3/045 ,  G06N3/047 ,  G06N3/084 ,  G06N20/00

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training neural networks. In one aspect, a system includes a neural network shrinking engine that is configured to receive a neural network being trained and generate a reduced neural network by a shrinking process. The shrinking process includes training the neural network based on a shrinking engine loss function that includes terms penalizing active neurons of the neural network and removing inactive neurons from the neural network. The system includes a neural network expansion engine that is configured to receive the neural network being trained and generate an expanded neural network by an expansion process including adding new neurons to the neural network and training the neural network based on an expanding engine loss function. The system includes a training subsystem that generates reduced neural networks and expanded neural networks.

公开(公告)号：US20190251445A1

公开(公告)日：2019-08-15

申请号：US15892890

申请日：2018-02-09

Applicant: Google LLC

Inventor： Yair Movshovitz-Attias ,  Elad Eban

IPC: G06N3/08 ,  G06N3/04

CPC classification number: G06N3/084 ,  G06N3/0445 ,  G06N3/063 ,  G06N3/08 ,  G06N7/005

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for neural network compression. In one aspect, a method comprises receiving a neural network and identifying a particular set of multiple weights of the neural network. Multiple anchor points are determined based on current values of the particular set of weights of the neural network. The neural network is trained by, at each of multiple training iterations, performing operations comprising adjusting the values of the particular set of weights by backpropagating gradients of a loss function. The loss function comprises a first loss function term based on a prediction accuracy of the neural network and a second loss function term based on a similarity of the current values of the particular set of weights to the anchor points. After training, the values of the particular set of weights are quantized based on the anchor points.

公开(公告)号：US20190147339A1

公开(公告)日：2019-05-16

申请号：US15813961

申请日：2017-11-15

Applicant: Google LLC

Inventor： Ofir Nachum ,  Ariel Gordon ,  Elad Eban ,  Bo Chen

IPC: G06N3/08 ,  G06N3/04 ,  G06F15/18

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training neural networks. In one aspect, a system includes a neural network shrinking engine that is configured to receive a neural network being trained and generate a reduced neural network by a shrinking process. The shrinking process includes training the neural network based on a shrinking engine loss function that includes terms penalizing active neurons of the neural network and removing inactive neurons from the neural network. The system includes a neural network expansion engine that is configured to receive the neural network being trained and generate an expanded neural network by an expansion process including adding new neurons to the neural network and training the neural network based on an expanding engine loss function. The system includes a training subsystem that generates reduced neural networks and expanded neural networks.

公开(公告)号：US20250061328A1

公开(公告)日：2025-02-20

申请号：US18723243

申请日：2022-12-15

Applicant: Google LLC

Inventor： Mariano Ruben Schain ,  Elad Eban

IPC: G06N3/08 ,  G06N3/0455

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing network inputs by applying augmentations to internal representations of the network inputs.

公开(公告)号：US20220215263A1

公开(公告)日：2022-07-07

申请号：US17701778

申请日：2022-03-23

Applicant: Google LLC

Inventor： Ofir Nachum ,  Ariel Gordon ,  Elad Eban ,  Bo Chen

IPC: G06N3/08 ,  G06N3/04 ,  G06N20/00

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training neural networks. In one aspect, a system includes a neural network shrinking engine that is configured to receive a neural network being trained and generate a reduced neural network by a shrinking process. The shrinking process includes training the neural network based on a shrinking engine loss function that includes terms penalizing active neurons of the neural network and removing inactive neurons from the neural network. The system includes a neural network expansion engine that is configured to receive the neural network being trained and generate an expanded neural network by an expansion process including adding new neurons to the neural network and training the neural network based on an expanding engine loss function. The system includes a training subsystem that generates reduced neural networks and expanded neural networks.

公开(公告)号：US11315019B2

公开(公告)日：2022-04-26

申请号：US15813961

申请日：2017-11-15

Applicant: Google LLC

Inventor： Ofir Nachum ,  Ariel Gordon ,  Elad Eban ,  Bo Chen

IPC: G06N3/08 ,  G06N3/04 ,  G06N20/00

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training neural networks. In one aspect, a system includes a neural network shrinking engine that is configured to receive a neural network being trained and generate a reduced neural network by a shrinking process. The shrinking process includes training the neural network based on a shrinking engine loss function that includes terms penalizing active neurons of the neural network and removing inactive neurons from the neural network. The system includes a neural network expansion engine that is configured to receive the neural network being trained and generate an expanded neural network by an expansion process including adding new neurons to the neural network and training the neural network based on an expanding engine loss function. The system includes a training subsystem that generates reduced neural networks and expanded neural networks.