Invention Grant
- Patent Title: Training a model using parameter server shards
-
Application No.: US15231534Application Date: 2016-08-08
-
Publication No.: US09721214B1Publication Date: 2017-08-01
- Inventor: Gregory S. Corrado , Kai Chen , Jeffrey A. Dean , Samy Bengio , Rajat Monga , Matthieu Devin
- Applicant: Google Inc.
- Applicant Address: US CA Mountain View
- Assignee: Google Inc.
- Current Assignee: Google Inc.
- Current Assignee Address: US CA Mountain View
- Agency: Fish & Richardson P.C.
- Main IPC: G06F15/18
- IPC: G06F15/18 ; G06N99/00 ; G06N7/08 ; G06N7/00 ; G06N5/02 ; G06K9/62

Abstract:
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a model using parameter server shards. One of the methods includes receiving, at a parameter server shard configured to maintain values of a disjoint partition of the parameters of the model, a succession of respective requests for parameter values from each of a plurality of replicas of the model; in response to each request, downloading a current value of each requested parameter to the replica from which the request was received; receiving a succession of uploads, each upload including respective delta values for each of the parameters in the partition maintained by the shard; and updating values of the parameters in the partition maintained by the parameter server shard repeatedly based on the uploads of delta values to generate current parameter values.
Information query