公开(公告)号：US10223544B1

公开(公告)日：2019-03-05

申请号：US15222435

申请日：2016-07-28

Applicant: EMC Corporation

Inventor： Prashant Pogde ,  Fabiano C. Botelho ,  Nitin Garg

IPC: G06F21/62

Abstract: In one embodiment, in response to a request received from a client for retrieving a data object stored in a storage system, a root key is obtained from the request. The data object is represented by metadata in a hierarchical structure having a plurality of levels. Each level includes a plurality of nodes and each node being one of a root node, a leaf node and an intermediate node. The hierarchical structure of metadata associated with the data object is traversed in a top-down approach to decrypt each of a plurality of nodes in the hierarchical structure using a key provided from its parent node, starting from the root node to the leaf nodes, including decrypting the root node using the root key. Decrypted data associated with the plurality of nodes is transmitted to the client.

公开(公告)号：US09460389B1

公开(公告)日：2016-10-04

申请号：US13907760

申请日：2013-05-31

Applicant: EMC Corporation

Inventor： Fabiano C. Botelho ,  Mark Chamness ,  Dmitry Serdyuk ,  Guilherme Menezes

IPC: G06F17/00 ,  G06N5/02

CPC classification number: G06N5/022 ,  G06F11/008 ,  G06F12/0253 ,  G06N5/025 ,  G06N99/005

Abstract: Mechanisms for predicting a GC duration are described herein. In one embodiment, the mechanisms include receiving a first set of features determined based on current operating status and prior garbage collection (GC) statistics of a first storage system. In one embodiment, the mechanisms include predicting a GC duration of a first GC process being performed at the first storage system by applying a predictive model on the first set of features, wherein the predictive model was generated based on a second set of features received periodically from a plurality of storage systems.

Abstract translation: 本文描述了用于预测GC持续时间的机制。 在一个实施例中，机制包括接收基于第一存储系统的当前操作状态和先前垃圾收集（GC）统计确定的第一组特征。 在一个实施例中，所述机制包括通过在第一组特征上应用预测模型来预测在第一存储系统执行的第一GC进程的GC持续时间，其中基于周期性地接收到的第二组特征来生成预测模型 从多个存储系统。

公开(公告)号：US09367448B1

公开(公告)日：2016-06-14

申请号：US13909875

申请日：2013-06-04

Applicant: EMC Corporation

Inventor： Fabiano C. Botelho ,  Dheer Moghe ,  Hung Hing (Anthony) Pang ,  Guilherme Vale Ferreira Menezes

IPC: G06F17/30 ,  G06F12/02

CPC classification number: G06F12/0253 ,  G06F3/0608 ,  G06F3/0623 ,  G06F3/0641 ,  G06F3/0652 ,  G06F3/0689

Abstract: A garbage collector of a storage system traverses a namespace of a file system of the storage system to verify data integrity of segments. The namespace identifies files that are represented by segments arranged in multiple levels in a hierarchy, where an upper level segment includes one or more references to one or more lower level segments, and at least one segment is referenced by multiple files. Traversing the namespace includes computing and verifying checksums all segments in a level-by-level manner, where checksums of an upper level are verified before any of checksums of a lower level are verified. Upon all checksums of all levels have been verified, a garbage collection process is performed on the segments stored in the storage system.

Abstract translation: 存储系统的垃圾回收器遍历存储系统的文件系统的命名空间，以验证段的数据完整性。 命名空间标识由层级中多个级别排列的片段表示的文件，其中上级段包括对一个或多个较低级别段的一个或多个引用，并且至少一个段由多个文件引用。 遍历命名空间包括以逐级方式计算和验证所有段的校验和，其中验证较低级别的任何校验和之前的上级校验和。 在所有级别的所有校验和已经被验证之后，对存储在存储系统中的段进行垃圾回收处理。

公开(公告)号：US09405761B1

公开(公告)日：2016-08-02

申请号：US14065812

申请日：2013-10-29

Applicant: EMC Corporation

Inventor： Fabiano C. Botelho ,  Hung Hing (Anthony) Pang

IPC: G06F17/30

CPC classification number: G06F17/30138 ,  G06F17/30091 ,  G06F17/30156 ,  G06F17/30371

Abstract: Exemplary methods for verifying data integrity for garbage collection with limited memory include maintaining a data structure that includes a plurality of entries, storing states of a group of segments compressed therein. In response to receiving a request for transitioning a segment from a first state to a second state, retrieving a first entry value of an entry associated with the first segment, generating a second entry value based on the first entry value, the first state, the second state, and a value obtained from a first lookup table based on the first segment. The methods also include writing back the second entry value to the first entry of the data structure. In one embodiment, in response to determining all entries of the data structure reach a predetermined final state, performing a garbage collection process on the segments stored in the storage system.

Abstract translation: 用于利用有限存储器验证垃圾收集的数据完整性的示例性方法包括维护包括多个条目的数据结构，存储在其中压缩的一组段的状态。 响应于接收到将段从第一状态转换到第二状态的请求，检索与第一段相关联的条目的第一条目值，基于第一条目值生成第二条目值，第一状态， 第二状态，以及基于第一段从第一查找表获得的值。 这些方法还包括将第二条目值写回数据结构的第一个条目。 在一个实施例中，响应于确定数据结构的所有条目达到预定的最终状态，对存储在存储系统中的段执行垃圾回收处理。

公开(公告)号：US09317218B1

公开(公告)日：2016-04-19

申请号：US13763522

申请日：2013-02-08

Applicant: EMC Corporation

Inventor： Fabiano C. Botelho ,  Nitin Garg ,  Philip N. Shilane ,  Grant Wallace

IPC: G06F12/00 ,  G06F3/06

CPC classification number: G06F3/0655 ,  G06F17/30156 ,  G06F17/30159 ,  G06F17/30949

Abstract: Techniques for sanitizing a storage system are described herein. In one embodiment, for each of fingerprints representing data chunks stored in a first container of the storage system, a lookup operation in a live bit vector based on the fingerprint is performed to determine whether a corresponding data chunk is live. In one embodiment, a bit in a copy bit vector corresponding to the data chunk is populated based on the lookup operation. In one embodiment, after all of the bits corresponding to the data chunks of the first container have been populated in the CBV, data chunks represented by the CBV are copied from the first container to a second container, and records of the data chunks in the first container are erased.

Abstract translation: 本文描述了用于消毒存储系统的技术。 在一个实施例中，对于代表存储在存储系统的第一容器中的数据块的每个指纹，执行基于指纹的实时比特向量中的查找操作，以确定对应的数据块是否是活的。 在一个实施例中，基于查找操作填充与数据块对应的复制位向量中的位。 在一个实施例中，在对应于第一容器的数据块的所有位已经被填充到CBV之后，由CBV表示的数据块从第一容器复制到第二容器，并将数据块的记录在 第一个容器被擦除。

公开(公告)号：US10019323B1

公开(公告)日：2018-07-10

申请号：US14225188

申请日：2014-03-25

Applicant: EMC Corporation

Inventor： Lan Bai ,  Fabiano C. Botelho ,  Mahesh Kamat ,  Nitin Garg

IPC: G06F17/30 ,  G06F11/14

CPC classification number: G06F11/1469 ,  G06F11/1004 ,  G06F11/1435 ,  G06F16/184

Abstract: A computer-implemented method is disclosed. The method starts with determining a first container of a storage system is invalid. The method continues with the storage system setting a data recovery state for the first container to be en-queue, which indicates that data of at least one of the data segments needs to be recovered from the first container, and executing a process to recover any container having an en-queue data recovery state, and for each of the containers, to recover any valid data segment from the corresponding container. The process includes scanning the data segments of the first container to find valid data segments, moving or replicating the valid data segments to a second container, and setting the data recovery state for the first container to be complete once all the valid data segments are moved or replicated to the second container.

公开(公告)号：US09424185B1

公开(公告)日：2016-08-23

申请号：US13909871

申请日：2013-06-04

Applicant: EMC Corporation

Inventor： Fabiano C. Botelho ,  Dheer Moghe ,  Guilherme Vale Ferreira Menezes ,  Nitin Garg ,  Srikant Varadan

IPC: G06F17/30 ,  G06F12/02

CPC classification number: G06F12/0253 ,  G06F2212/401

Abstract: A garbage collector of a storage system traverses a namespace of a file system of the storage system to identify segments that are alive in a breadth-first manner. The namespace includes information identifying files that are represented by segments arranged in a plurality of levels in a hierarchy, where an upper level segment includes one or more references to one or more lower level segments, and at least one segment is referenced by multiple files. All live segments of an upper level are identified before any of live segments of a lower level are identified. Upon all live segments of all levels have been identified, the live segments are copied from their original storage locations to a new storage location, and a storage space associated with the original storage locations is reclaimed.

Abstract translation: 存储系统的垃圾回收器遍历存储系统的文件系统的命名空间以识别以广度优先的方式活着的段。 命名空间包括标识由分层中多个级别排列的段表示的文件的信息，其中上级段包括对一个或多个较低级别段的一个或多个引用，并且至少一个段由多个文件引用。 识别较低级别的任何实时段之前，上级的所有实时段都被识别。 在所有级别的所有实时段都已被识别后，实时段将从其原始存储位置复制到新的存储位置，并且回收与原始存储位置相关联的存储空间。

公开(公告)号：US11151030B1

公开(公告)日：2021-10-19

申请号：US15252214

申请日：2016-08-31

Applicant: EMC Corporation

Inventor： Fabiano C. Botelho ,  Mark Chamness ,  Dmitry Serdyuk ,  Guilherme Menezes

IPC: G06N20/20 ,  G06F12/02 ,  G06F3/06 ,  G06N20/10

Abstract: A first set of garbage collection (GC) features and non-GC features associated with a storage system are received, the first set of features being associated with a predetermined start date and a time window. A learning equation is generated having a plurality of vectors of GC features and a plurality of vectors of non-GC features. For a current iteration representing a current GC process, it is determined whether a first prior GC process was started within the time window. An entry of vectors of the non-GC features of the learning equation is populated based on corresponding feature values of the first set of non-GC features, in response to determining that the first prior GC process was started within the time window. A predetermined regression algorithm is applied to the learning equation to generate a GC duration predictive model to predict a GC duration of a subsequent GC process.

公开(公告)号：US09432192B1

公开(公告)日：2016-08-30

申请号：US14229364

申请日：2014-03-28

Applicant: EMC Corporation

Inventor： Prashant Pogde ,  Fabiano C. Botelho ,  Nitin Garg

IPC: H04K1/00 ,  H04L9/30

CPC classification number: H04L9/0836 ,  H04L9/0891 ,  H04L9/0894 ,  H04L2209/60

Abstract: In one embodiment, metadata of a data object to be stored in a storage system is received, where the metadata is in a hierarchical structure having multiple levels, each level having multiple nodes and each node being one of a root node, a leaf node and an intermediate node. Each leaf node represents a deduplicated segment associated with the data object. The hierarchical structure is traversed to encrypt each of the nodes in a bottom-up approach, starting from leaf nodes, using different keys. A child key for encrypting content of a child node is stored in a parent node that references the child node, and the child key is encrypted by a parent key associated with the parent node. The encrypted content of the nodes are then stored in one or more storage units of the storage system in a deduplicated manner.

Abstract translation: 在一个实施例中，接收要存储在存储系统中的数据对象的元数据，其中元数据处于具有多个级别的分层结构中，每个级别具有多个节点，并且每个节点是根节点，叶节点和 一个中间节点。 每个叶节点表示与数据对象相关联的重复数据删除段。 遍历层次结构以自下而上的方式加密每个节点，从叶节点开始，使用不同的密钥。 用于加密子节点的内容的子密钥存储在引用子节点的父节点中，并且子密钥由与父节点相关联的父密钥加密。 然后将节点的加密内容以重复数据删除的方式存储在存储系统的一个或多个存储单元中。

公开(公告)号：US09430164B1

公开(公告)日：2016-08-30

申请号：US13763508

申请日：2013-02-08

Applicant: EMC Corporation

Inventor： Fabiano C. Botelho ,  Nitin Garg ,  Philip N. Shilane ,  Grant Wallace

IPC: G06F12/00 ,  G06F3/06 ,  G06F12/08

CPC classification number: G06F3/0655 ,  G06F3/0608 ,  G06F3/0641 ,  G06F3/0652 ,  G06F3/067 ,  G06F11/14 ,  G06F11/1453 ,  G06F12/0862 ,  G06F2201/83 ,  G06F2212/466

Abstract: Techniques for sanitizing a storage system are described herein. In one embodiment, for each file stored in the storage system, a list of fingerprints representing data chunks of the file is obtained. In such an embodiment, for each of the fingerprints, identifying a first container storing a data chunk corresponding to the fingerprint is identified, and determining a storage location of the first container in which the data chunk is stored is determined. In one embodiment, a bit in copy bit vector (CBV) is populated based on the identified container and the storage location. In one embodiment, after all of the bits corresponding to the data chunks of the first container have been populated in the CBV, data chunks represented by the CBV are copied from the first container to a second container, and records of the data chunks in the first container are erased.

Abstract translation: 本文描述了用于消毒存储系统的技术。 在一个实施例中，对于存储在存储系统中的每个文件，获得表示文件的数据块的指纹列表。 在这样的实施例中，对于每个指纹，识别存储与指纹相对应的数据块的第一容器，并确定存储数据块的第一容器的存储位置。 在一个实施例中，基于所识别的容器和存储位置来填充复制位矢量（CBV）中的位。 在一个实施例中，在对应于第一容器的数据块的所有位已经被填充到CBV之后，由CBV表示的数据块从第一容器复制到第二容器，并将数据块的记录在 第一个容器被擦除。