公开(公告)号：US11775823B2

公开(公告)日：2023-10-03

申请号：US17014139

申请日：2020-09-08

Applicant: Google LLC

Inventor： Sashank Jakkam Reddi ,  Sanjiv Kumar ,  Manzil Zaheer ,  Satyen Chandrakant Kale

IPC: G06N3/08 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/045

Abstract: Generally, the present disclosure is directed to systems and methods that perform adaptive optimization with improved convergence properties. The adaptive optimization techniques described herein are useful in various optimization scenarios, including, for example, training a machine-learned model such as, for example, a neural network. In particular, according to one aspect of the present disclosure, a system implementing the adaptive optimization technique can, over a plurality of iterations, employ an adaptive effective learning rate while also ensuring that the effective learning rate is non-increasing.

公开(公告)号：US11586904B2

公开(公告)日：2023-02-21

申请号：US16130058

申请日：2018-09-13

Applicant: Google LLC

Inventor： Sashank Jakkam Reddi ,  Sanjiv Kumar ,  Satyen Chandrakant Kale

IPC: G06N3/08 ,  G06N3/04 ,  G06F17/16 ,  G06N20/00

Abstract: Generally, the present disclosure is directed to systems and methods that perform adaptive optimization with improved convergence properties. The adaptive optimization techniques described herein are useful in various optimization scenarios, including, for example, training a machine-learned model such as, for example, a neural network. In particular, according to one aspect of the present disclosure, a system implementing the adaptive optimization technique can, over a plurality of iterations, employ an adaptive learning rate while also ensuring that the learning rate is non-increasing.

公开(公告)号：US10769529B2

公开(公告)日：2020-09-08

申请号：US16657356

申请日：2019-10-18

Applicant: Google LLC

Inventor： Sashank Jakkam Reddi ,  Sanjiv Kumar ,  Manzil Zaheer ,  Satyen Chandrakant Kale

IPC: G06N3/04 ,  G06N3/08

Abstract: Generally, the present disclosure is directed to systems and methods that perform adaptive optimization with improved convergence properties. The adaptive optimization techniques described herein are useful in various optimization scenarios, including, for example, training a machine-learned model such as, for example, a neural network. In particular, according to one aspect of the present disclosure, a system implementing the adaptive optimization technique can, over a plurality of iterations, employ an adaptive effective learning rate while also ensuring that the effective learning rate is non-increasing.

公开(公告)号：US20200090031A1

公开(公告)日：2020-03-19

申请号：US16130058

申请日：2018-09-13

Applicant: Google LLC

Inventor： Sashank Jakkam Reddi ,  Sanjiv Kumar ,  Satyen Chandrakant Kale

IPC: G06N3/08 ,  G06F15/18 ,  G06F17/16 ,  G06N3/04

Abstract: Generally, the present disclosure is directed to systems and methods that perform adaptive optimization with improved convergence properties. The adaptive optimization techniques described herein are useful in various optimization scenarios, including, for example, training a machine-learned model such as, for example, a neural network. In particular, according to one aspect of the present disclosure, a system implementing the adaptive optimization technique can, over a plurality of iterations, employ an adaptive learning rate while also ensuring that the learning rate is non-increasing.

公开(公告)号：US12271810B2

公开(公告)日：2025-04-08

申请号：US17100253

申请日：2020-11-20

Applicant: Google LLC

Inventor： Sashank Jakkam Reddi ,  Sanjiv Kumar ,  Manzil Zaheer ,  Zachary Burr Charles ,  Zachary Alan Garrett ,  John Keith Rush ,  Jakub Konecny ,  Hugh Brendan McMahan

IPC: G06N3/08 ,  G06N3/04 ,  G06N3/045

Abstract: A computing system and method can be used to implement a version of federated learning (FL) that incorporates adaptivity (e.g., leverages an adaptive learning rate). In particular, the present disclosure provides a general optimization framework in which (1) clients perform multiple epochs of training using a client optimizer to minimize loss on their local data and (2) a server system updates its global model by applying a gradient-based server optimizer to the average of the clients' model updates. This framework can seamlessly incorporate adaptivity by using adaptive optimizers as client and/or server optimizers. Building upon this general framework, the present disclosure also provides example specific adaptive optimization techniques for FL which use per-coordinate methods as server optimizers. By focusing on adaptive server optimization, the use of adaptive learning rates is enabled without increase in client storage or communication costs and compatibility with cross-device FL can be ensured.

公开(公告)号：US20230394310A1

公开(公告)日：2023-12-07

申请号：US18453837

申请日：2023-08-22

Applicant: Google LLC

Inventor： Sashank Jakkam Reddi ,  Sanjiv Kumar ,  Manzil Zaheer ,  Satyen Chandrakant Kale

IPC: G06N3/08 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/045

Abstract: Generally, the present disclosure is directed to systems and methods that perform adaptive optimization with improved convergence properties. The adaptive optimization techniques described herein are useful in various optimization scenarios, including, for example, training a machine-learned model such as, for example, a neural network. In particular, according to one aspect of the present disclosure, a system implementing the adaptive optimization technique can, over a plurality of iterations, employ an adaptive effective learning rate while also ensuring that the effective learning rate is non-increasing.

公开(公告)号：US11676033B1

公开(公告)日：2023-06-13

申请号：US16812160

申请日：2020-03-06

Applicant: Google LLC

Inventor： Aditya Krishna Menon ,  Ankit Singh Rawat ,  Sashank Jakkam Reddi ,  Sanjiv Kumar

IPC: G06N3/04 ,  G06N3/084

CPC classification number: G06N3/084 ,  G06N3/04

Abstract: A method for training a machine learning model, e.g., a neural network, using a regularization scheme is disclosed. The method includes generating regularized partial gradients of losses computed using an objective function for training the machine learning model.

公开(公告)号：US20230113984A1

公开(公告)日：2023-04-13

申请号：US18081403

申请日：2022-12-14

Applicant: Google LLC

Inventor： Sashank Jakkam Reddi ,  Sanjiv Kumar ,  Satyen Chandrakant Kale

IPC: G06N3/08 ,  G06F17/16 ,  G06N20/00 ,  G06N3/045

Abstract: Generally, the present disclosure is directed to systems and methods that perform adaptive optimization with improved convergence properties. The adaptive optimization techniques described herein are useful in various optimization scenarios, including, for example, training a machine-learned model such as, for example, a neural network. In particular, according to one aspect of the present disclosure, a system implementing the adaptive optimization technique can, over a plurality of iterations, employ an adaptive learning rate while also ensuring that the learning rate is non-increasing.

公开(公告)号：US20210049298A1

公开(公告)日：2021-02-18

申请号：US16994396

申请日：2020-08-14

Applicant: Google LLC

Inventor： Ananda Theertha Suresh ,  Xinnan Yu ,  Sanjiv Kumar ,  Sashank Jakkam Reddi ,  Venkatadheeraj Pichapati

IPC: G06F21/62 ,  G06K9/62 ,  G06N3/04 ,  G06F17/17

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for privacy preserving training of a machine learning model.

公开(公告)号：US12229675B2

公开(公告)日：2025-02-18

申请号：US18081403

申请日：2022-12-14

Applicant: Google LLC

Inventor： Sashank Jakkam Reddi ,  Sanjiv Kumar ,  Satyen Chandrakant Kale

IPC: G06N3/08 ,  G06F17/16 ,  G06N3/045 ,  G06N20/00

Abstract: Generally, the present disclosure is directed to systems and methods that perform adaptive optimization with improved convergence properties. The adaptive optimization techniques described herein are useful in various optimization scenarios, including, for example, training a machine-learned model such as, for example, a neural network. In particular, according to one aspect of the present disclosure, a system implementing the adaptive optimization technique can, over a plurality of iterations, employ an adaptive learning rate while also ensuring that the learning rate is non-increasing.