-
公开(公告)号:US11113161B2
公开(公告)日:2021-09-07
申请号:US16114033
申请日:2018-08-27
Applicant: Amazon Technologies, Inc.
Inventor: Colin Laird Lazier , Adam Frederick Brock
Abstract: A data transfer device is used to augment the capabilities of a data storage system. The data transfer device may be capable of persistently storing data for an indeterminate amount of time, and may be configured to store a portion of a bundle of redundancy coded shards that span between the data transfer device and a data storage system configured to store the remainder of the bundle. Data stored on the data transfer device may be read from and written directly to the data transfer device without transfer of data to the data storage system. If the data transfer device is not available, the remaining shards of the bundle may provide a regenerated, original form of the data.
-
公开(公告)号:US10970276B2
公开(公告)日:2021-04-06
申请号:US16105258
申请日:2018-08-20
Applicant: Amazon Technologies, Inc.
Inventor: Colin Laird Lazier , Brian J. Deen
IPC: G06F16/215 , G06F16/907 , G06F16/24 , G06F11/20 , G06F11/14 , G06F11/34
Abstract: In response to receiving a request from a client to store an object, a key-durable storage system may assign the object to a volume in its data store, generate a key for the object (e.g., an opaque identifier that encodes information for locating the object in the data store), store the object on one disk in the assigned volume, store the key redundantly in the assigned volume (e.g., using a replication or erasure coding technique), and may return the key to the client. To retrieve the object, the client may send a request including the key, and the system may return the object to the client. If a disk fails, the system may determine which objects were lost, and may return the corresponding keys to the appropriate clients in a notification. The system may be used to back up a more expensive object-redundant storage system.
-
公开(公告)号:US20200379651A1
公开(公告)日:2020-12-03
申请号:US16998709
申请日:2020-08-20
Applicant: Amazon Technologies, Inc.
Inventor: Colin Laird Lazier
IPC: G06F3/06
Abstract: A device such as a network-attachable data transfer device may be configured to operate in a cluster to coordinate the storage of data. A first manifest may be generated inventorying a first set of data successfully transferred to the data transfer device from a data source. A second manifest may be generated inventorying a second set of data successfully transferred from the data transfer device to a data destination. The first manifest may be compared with the second manifest to determine a transfer status of one or more data objects. The transfer status may indicate one or more data objects successfully transferred to the data destination from the data source. The one or more objects may be processed according to the transfer status.
-
公开(公告)号:US10732902B1
公开(公告)日:2020-08-04
申请号:US16143258
申请日:2018-09-26
Applicant: Amazon Technologies, Inc.
Inventor: Colin Laird Lazier , Marco Argenti
Abstract: A hub device or edge device implements adaptive data compression. The model training service analyzes the received data. A machine learning model of the hub device receives time-series data from one or more data sources and classifies respective portions of the time-series data as respective patterns. A data compressor at the hub device generates compressed data by applying different compression techniques to the respective portions of the time-series data according to a mapping of the compression techniques to the respective patterns. The hub device then transmits the compressed data to an endpoint for processing (e.g., another device that uses the compressed data). The hub device receives feedback for the compressed data. In response to the feedback, the hub device changes one or more of the compression techniques that are mapped to the respective patterns.
-
公开(公告)号:US10521356B1
公开(公告)日:2019-12-31
申请号:US13770892
申请日:2013-02-19
Applicant: Amazon Technologies, Inc.
Inventor: Colin Laird Lazier
IPC: G06F11/10 , G06F12/1018 , G06F3/06
Abstract: Components of a data object are distributed throughout a data storage system. Manifests are used to store the locations of the components of data objects in a data storage system to allow for subsequent reconstruction of the data objects. The manifests may be stored in another data storage system when cost projections indicate it being economical to do so. If a manifest for a data object becomes lost or otherwise inaccessible, clues are used to regenerate the manifest, thereby providing a continued ability to access the components of the data object to reconstruct the data object.
-
公开(公告)号:US20190340571A1
公开(公告)日:2019-11-07
申请号:US16517311
申请日:2019-07-19
Applicant: Amazon Technologies, Inc.
Inventor: Colin Laird Lazier , William J. Vass
Abstract: A network-attachable data transfer device housed within a shippable enclosure that incorporates an updateable electronic display for displaying shipping destination information is disclosed. The device may be initialized (e.g., prepared to receive data, and the updateable electronic shipping display set to the shipping destination) by a service provider and shipped, in accordance with the displayed destination address, as a self-contained shipping unit. The device may be installed onto a network at the destination and loaded with data. The display may also be updated with the next destination address such that the device is shipped to the updated destination address (e.g., back to the service provider, or onto other destinations before being send back to the service provider). When the device is received back at the service provider, the data is transferred from the device to a service provider storage facility, wiped of data, and prepared to be sent out again.
-
公开(公告)号:US10394762B1
公开(公告)日:2019-08-27
申请号:US14789837
申请日:2015-07-01
Applicant: Amazon Technologies, Inc.
Inventor: Paul David Franklin , James Caleb Kirschner , Bryan James Donlan , Colin Laird Lazier
Abstract: Techniques for encoding data in grid encoded data storage systems are described herein. Data is stored in a grid of shards using grid encoding techniques that store the data in a combination of data shards and derived shards. Because the data is redundantly encoded in the grid of shards, a plurality of distinct copies of the data are reproducible from the grid of shards where each distinct copy is reproducible from a non-overlapping set of shards of the grid of shards.
-
公开(公告)号:US10366062B1
公开(公告)日:2019-07-30
申请号:US15083125
申请日:2016-03-28
Applicant: Amazon Technologies, Inc.
Inventor: Colin Laird Lazier , Adam Frederick Brock , James Caleb Kirschner
Abstract: A cluster of data transfer devices is used to augment the capabilities of a data storage system. For example, the cluster of data transfer devices may be configured to store a portion of a bundle of redundancy coded shards in a similar fashion as a data storage system. As another example, the cluster may be configured to provide other capabilities incident to the devices used, such as computational capabilities. Data stored on the cluster may be read from and written directly to the cluster without transfer of data to the data storage system. In some embodiments, a connecting entity (such as a customer entity) may interchangeably interface with the data storage system and the cluster, and the requested capabilities may be directed to either in a fashion that is transparent to the requestor.
-
公开(公告)号:US10270476B1
公开(公告)日:2019-04-23
申请号:US14741407
申请日:2015-06-16
Applicant: Amazon Technologies, Inc.
Inventor: Colin Laird Lazier
Abstract: Techniques and methods for generating and implementing multiple layers of redundancy coded data are disclosed. For example, a redundancy coding scheme may include data elements that include data that is unencoded relative to the input, yet may still fully participate in providing redundancy to any data element in a given set. In a layered scheme, the input may include a bundle or group of encoded (or unencoded) data elements, thereby nesting two or more layers of redundancy coding. The specific amount of redundancy generated by such a scheme may be adjusted and adapted to failure characteristics of the entity on which the data elements are stored.
-
公开(公告)号:US10162709B1
公开(公告)日:2018-12-25
申请号:US15360864
申请日:2016-11-23
Applicant: Amazon Technologies, Inc.
Inventor: Colin Laird Lazier , Masataka Kubo , Bryan James Donlan
Abstract: Techniques for storing incremental backups in long-term storage are described herein. A backup data set is generated from a set of data to backup and a deletion time for that backup data set is determined. A scheduled time period to perform the backup is selected based at least in part on a requested time period to perform the backup. The requested time period is altered if the performing the backup at the requested time period would produce a period of high resource contention at the destination for the backup. The backup is then stored at the destination at the scheduled time.
-
-
-
-
-
-
-
-
-