-
公开(公告)号:US11379332B2
公开(公告)日:2022-07-05
申请号:US17006514
申请日:2020-08-28
Applicant: Amazon Technologies, Inc.
Inventor: Swaminathan Sivasubramanian , Grant Alexander Macdonald McAlister , Paul David Franklin , Rajesh Sudhakar Sheth , James Horsley
Abstract: Aspects of a data environment, such as the creation, provisioning, and management of data stores and instances, are managed using a separate control environment. A user can call into an externally-facing interface of the control environment, the call being analyzed to determine actions to be performed in the data environment. A monitoring component of the control plane also can periodically communicate with the data environment to determine any necessary actions to be performed, such as to recover from faults or events in the data environment. A workflow can be instantiated that includes tasks necessary to perform the action. For each task, state information can be passed to a component in the data environment operable to perform the task, until all tasks for an action are completed. Data in the data environment can be accessed directly using an externally-facing interface of the data environment, without accessing the control plane.
-
公开(公告)号:US10592336B1
公开(公告)日:2020-03-17
申请号:US15080503
申请日:2016-03-24
Applicant: Amazon Technologies, Inc.
Inventor: Bryan James Donlan , Paul David Franklin , James Caleb Kirschner
Abstract: A multilayered index is implemented for data stored in a data storage system. The multilayered index may include a granular index that specifies the location of specific files or abstractions within the data storage system, while a skip table specifies segments of the abstractions, and may point to both the granular index and the data storage devices of the data storage system. In redundancy coded systems, the use of multilayered indices may enable asynchronous retrieval of data during normal or adverse operation (such as during periods of varying data availability).
-
公开(公告)号:US10180912B1
公开(公告)日:2019-01-15
申请号:US14973677
申请日:2015-12-17
Applicant: Amazon Technologies, Inc.
Inventor: Paul David Franklin , Bryan James Donlan , Marvin Michael Theimer
Abstract: A computer system, such as a data storage system, implements techniques for segregating and controlling access to data stored in multiple regions. In some embodiments, redundancy coded shards generated from the data and stored in durable storage of a data storage system is allocated across multiple regions, but in a fashion that prevents actors with access to regions outside that of a “home” region from recovering a sufficient number of unique shards to regenerate the data represented thereby. In some embodiments, encryption is used to segregate the data by encrypting the generated shards, then storing the cryptographic information on or otherwise controlling access on hosts or other devices of only the home region.
-
公开(公告)号:US20180288049A1
公开(公告)日:2018-10-04
申请号:US15471941
申请日:2017-03-28
Applicant: Amazon Technologies, Inc.
Inventor: Bryan James Donlan , Paul David Franklin
IPC: H04L29/06 , H04L29/08 , H04L9/32 , H04L9/08 , G06F3/06 , G06F13/10 , G06F13/16 , G06F13/40 , G06F13/42
Abstract: A switching device is implemented in a network-attachable data transfer device to provide data storage access to other such devices. In some embodiments, network-attachable data transfer devices are arranged in a clustered configuration to provide various computational and storage services. When one or more devices of the cluster fails, various implementations associated with the switching device, via an external data interface, provide operational mitigation, optimized data recovery, and efficient reinstatement of normal operation of the cluster.
-
公开(公告)号:US10042848B1
公开(公告)日:2018-08-07
申请号:US14578185
申请日:2014-12-19
Applicant: Amazon Technologies, Inc.
Inventor: Bryan James Donlan , Paul David Franklin
Abstract: Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored.
-
6.
公开(公告)号:US09158927B1
公开(公告)日:2015-10-13
申请号:US13925497
申请日:2013-06-24
Applicant: Amazon Technologies, Inc.
Inventor: Paul David Franklin , Bryan James Donlan
CPC classification number: G06F21/602
Abstract: Reliable and efficient storage and reconstruction of secure data files is provided. Encrypted fragments are generated by exclusive-OR (XOR) based erasure-encoding and XOR encryption of data files. At least some of the encrypted fragments, and preferably at least two copies of such encrypted fragments, are stored at two or more locations, such as but not limited to two or more servers in two or more regional storage systems. Fragments are retrieved from one or more of the multiple locations and the original data file is reconstructed, even if different encryption techniques have been used. If not enough valid fragments from that original data file can be identified then hash values, checksums, seeds, and other techniques may be used to distinguish files and to identify related or identical files which may be used to reconstruct the data file.
Abstract translation: 提供安全数据文件的可靠和高效的存储和重建。 加密的片段通过基于异或(XOR)的数据文件的擦除编码和异或加密生成。 至少一些加密的片段,并且优选地,这种加密片段的至少两个副本存储在两个或更多个位置,例如但不限于两个或更多个区域存储系统中的两个或更多个服务器。 从多个位置中的一个或多个检索片段,并且即使使用不同的加密技术,也重构原始数据文件。 如果不能识别出原始数据文件中足够的有效片段,则可以使用散列值,校验和,种子和其他技术区分文件,并识别可用于重建数据文件的相关或相同的文件。
-
公开(公告)号:US11356445B2
公开(公告)日:2022-06-07
申请号:US15471941
申请日:2017-03-28
Applicant: Amazon Technologies, Inc.
Inventor: Bryan James Donlan , Paul David Franklin
Abstract: A switching device is implemented in a network-attachable data transfer device to provide data storage access to other such devices. In some embodiments, network-attachable data transfer devices are arranged in a clustered configuration to provide various computational and storage services. When one or more devices of the cluster fails, various implementations associated with the switching device, via an external data interface, provide operational mitigation, optimized data recovery, and efficient reinstatement of normal operation of the cluster.
-
公开(公告)号:US10860533B1
公开(公告)日:2020-12-08
申请号:US15631876
申请日:2017-06-23
Applicant: AMAZON TECHNOLOGIES, INC.
Inventor: Eric Neilsen , Paul David Franklin
IPC: G06F16/14 , G06F16/174
Abstract: A processing device receives a request to identify whether a file is a complete file. The processing device determines a first size of the file. The processing device determines whether the first size satisfies a size criterion. The processing device determines that the file is a complete file responsive to the determination that the first size satisfies the size criterion.
-
公开(公告)号:US10534669B1
公开(公告)日:2020-01-14
申请号:US16020825
申请日:2018-06-27
Applicant: Amazon Technologies, Inc.
Inventor: Seth William Markle , Bryan James Donlan , Paul David Franklin , Colin Laird Lazier
Abstract: A data storage service stores a dataset on a set of storage nodes in accordance with a first encoding. A set of shards constituting quorum, and one or more additional shards, are stored on the storage nodes. The data storage system determines to store the dataset according to a second encoding, in which the second encoding has a greater number of shards. The data storage system reconfigures the storage of the dataset in accordance with the second encoding, such that the reconfigured storage forms additional shards for the second encoding by combining portions of shards of the first encoding.
-
公开(公告)号:US10235402B1
公开(公告)日:2019-03-19
申请号:US14973712
申请日:2015-12-17
Applicant: Amazon Technologies, Inc.
IPC: G06F17/30
Abstract: One or more grids of redundancy coded shards, such as those stored or otherwise represented on grid encoded storage systems, are combinable or extensible. For example, a generator matrix of a redundancy code may be configured so as to have a sufficient number of fields to generate a grid. The generator matrix may initially be used to generate smaller grids, which can be combined into the target grid without re-encoding most or all of the data represented thereon. In some cases, vertically derived shards of the input grids may be combined using, e.g., matrix addition, which may then be directly allocated to the target grid, while data shards and horizontally derived shards may be allocated to the target grid with no further transformation.
-
-
-
-
-
-
-
-
-