Abstract:
Methods and systems for detecting error in data storage entities based at least in part on importance of data stored in the data storage entities. In an embodiment, multiple verification passes may be performed on a data storage entity comprising one or more data blocks. Each data block may be associated with a probability indicating the likelihood that the data block is to be selected for verification. During each verification pass, a subset of the data blocks may be selected based at least in part on the probabilities associated with the data blocks. The probabilities may be adjusted, for example, at the end of a verification pass, based on importance factors such as usage and verification information associated with the data blocks. The probabilities may be updated to facilitate timely detection of important data blocks. Additionally, error mitigation and/or correction routines may be performed in light of detected errors.
Abstract:
A snapshot depth for a volume is obtained, with the snapshot depth indicating an upper limit to a number of snapshot updates, where an individual snapshot update of the snapshot updates includes data of the volume that has been modified since a time of storage of a previous snapshot update and an amount, based at least in part on the snapshot depth, of other data of the volume. The number of snapshot updates sufficient to construct a full snapshot of a state of the volume at a particular point in time is determined based at least in part on the snapshot depth, and at least the number of snapshot updates is caused to be stored.
Abstract:
A block-based storage system may implement efficient replication for restoring a data volume from a reduced durability state. A storage node that is not replicating write requests for a data volume may determine that replication for the data volume is to be enabled. A peer storage node may be identified that maintains a stale replica of the data volume. One or more replication operations may be performed to update stale data chunks in the stale replica of the data volume with current data chunks without updating data chunks in the stale replica of the data volume that are current. Stale replicas that are no longer needed may be deleted according timeouts or the amount of stale data in the replica.
Abstract:
A block-based storage system may implement efficient replication for restoring a data volume from a reduced durability state. A storage node that is not replicating write requests for a data volume may determine that replication for the data volume is to be enabled. A peer storage node may be identified that maintains a stale replica of the data volume. One or more replication operations may be performed to update stale data chunks in the stale replica of the data volume with current data chunks without updating data chunks in the stale replica of the data volume that are current. Stale replicas that are no longer needed may be deleted according timeouts or the amount of stale data in the replica.
Abstract:
Write optimization for block-based storage performing snapshot operations may be implemented. Write requests for a particular data volume may be received for which a snapshot operation is in progress. A determination may be made as to whether a data chunk of the data volume modified as part of the write request has not yet been stored to a remote snapshot data store as part of the snapshot operation. For a data chunk that is to be modified and that has not yet been stored, the data chunk may be stored in a local in-memory volume snapshot buffer. Once the data chunk is stored in the in-memory volume snapshot buffer, the write request may be performed and acknowledged as complete. The data chunk may be sent to the remote snapshot data store asynchronously with regard to the acknowledgment of the write request.
Abstract:
A snapshot depth for a volume is obtained, with the snapshot depth indicating an upper limit to a number of snapshot updates, where an individual snapshot update of the snapshot updates includes data of the volume that has been modified since a time of storage of a previous snapshot update and an amount, based at least in part on the snapshot depth, of other data of the volume. The number of snapshot updates sufficient to construct a full snapshot of a state of the volume at a particular point in time is determined based at least in part on the snapshot depth, and at least the number of snapshot updates is caused to be stored.
Abstract:
A block-based storage system may implement dynamic durability adjustment for page cache write logging. A rate of incoming write requests for data volumes maintained at a storage node may be monitored. Based, at least in part, on the rate of incoming write requests, a dynamic modification to a durability property for a data volume may be made, such as enabling page cache write logging the data volume or disabling write logging for the data volume. When incoming write requests are received, a determination may be made as to whether page cache write logging for a particular data volume is enabled. For write requests with disabled page cache write logging, the page cache may be updated and the write request may be acknowledged without storing a log record describing the update in a page cache write log.
Abstract:
A block-based storage system may implement dynamic durability adjustment for page cache write logging. A rate of incoming write requests for data volumes maintained at a storage node may be monitored. Based, at least in part, on the rate of incoming write requests, a dynamic modification to a durability property for a data volume may be made, such as enabling page cache write logging the data volume or disabling write logging for the data volume. When incoming write requests are received, a determination may be made as to whether page cache write logging for a particular data volume is enabled. For write requests with disabled page cache write logging, the page cache may be updated and the write request may be acknowledged without storing a log record describing the update in a page cache write log.
Abstract:
Methods and systems for detecting error in data storage entities based at least in part on importance of data stored in the data storage entities. In an embodiment, multiple verification passes may be performed on a data storage entity comprising one or more data blocks. Each data block may be associated with a probability indicating the likelihood that the data block is to be selected for verification. During each verification pass, a subset of the data blocks may be selected based at least in part on the probabilities associated with the data blocks. The probabilities may be adjusted, for example, at the end of a verification pass, based on importance factors such as usage and verification information associated with the data blocks. The probabilities may be updated to facilitate timely detection of important data blocks. Additionally, error mitigation and/or correction routines may be performed in light of detected errors.