Abstract:
A switching device is implemented in a network-attachable data transfer device to provide data storage access to other such devices. In some embodiments, network-attachable data transfer devices are arranged in a clustered configuration to provide various computational and storage services. When one or more devices of the cluster fails, various implementations associated with the switching device, via an external data interface, provide operational mitigation, optimized data recovery, and efficient reinstatement of normal operation of the cluster.
Abstract:
Techniques for incrementally updating grid encoding data storage systems are described herein. A grid of shards with a plurality of virtual shards is created where each virtual shard is a representation of a shard in the grid of shards that is not backed by a data storage device and where each shard of the grid of shards has an index value. Data is then stored in the grid of shards by updating a shard to store the data and by also updating a second shard based on a set of shards with the same index value as the shard updated to store the data.
Abstract:
Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored.
Abstract:
A data center may include a tape library rack module along with rack computer systems. The rack computer systems may be configured to provide computing capacity within a data center environment. In some embodiments, the tape library rack module may include an enclosure encompassing an interior of the tape library rack module, a rack within the interior, and a tape library unit mounted on the rack. The tape library rack unit may include tape cartridges configured to store data within a tape environment that is different than the data center environment. The tape library rack unit may be within a portion of the interior that is enclosed such that it is environmentally isolated from the data center environment. In some examples, the tape library rack module may include a cooling unit and/or a humidifier unit, which may provide the tape environment to the environmentally isolated portion of the interior of the tape library rack module.
Abstract:
A data storage service distributes a plurality of data fragments corresponding to a data object among one or more data storage host groupings in a manner that avoids a possibility of correlated loss of multiple data fragments by consolidation of data of a data storage host grouping onto a single data storage host. The data storage service selects a data storage host grouping and determines an amount of used capacity for the selected data storage host grouping. If the selected grouping satisfies an emptiness threshold, the data storage service selects a data storage host from the grouping and consolidates one or more data sets of the grouping onto the selected data storage host. Subsequently, the data storage service updates metadata for each data storage host of the selected data storage host grouping to specify a location of data stored therein.
Abstract:
Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored. The volumes may be apportioned into failure-decorrelated subsets, and archives stored thereto may be apportioned to such subsets.
Abstract:
Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored. The volumes may be apportioned into failure-decorrelated subsets, and archives stored thereto may be apportioned to such subsets.
Abstract:
Erasure encoded fragments may be originally generated by applying an erasure encoding scheme to a data file. An erasure encoded fragment may be subsequently generated directly from previously generated erasure encoded fragments or by reconstructing the original data file and then erasure encoding the reconstructed data file. The integrity or fidelity of such a subsequently generated erasure encoded fragment may be verified by newly generating an error detection code, such as but not limited to a checksum, for the subsequently generated erasure encoded fragment, and comparing that subsequently generated error detection code against an error detection code previously generated for a previous or original version of the erasure encoded fragment. Each error detection code is preferably stored in association with its corresponding erasure encoded fragment and with one or more other erasure encoded fragments. Thus, each error detection code is saved in at least two locations.
Abstract:
Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored. The volumes may be apportioned into failure-decorrelated subsets, and archives stored thereto may be apportioned to such subsets.
Abstract:
Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored.