Storage scheme for a distributed storage system

    公开(公告)号:US10579276B2

    公开(公告)日:2020-03-03

    申请号:US15703803

    申请日:2017-09-13

    IPC分类号: G06F12/00 G06F3/06

    摘要: A storage scheme allocates portions of a logical volume to storage nodes in excess of the capacity of the storage nodes. Slices of the storage nodes and segments of slices are allocated in response to write requests such that actual allocation on the storage nodes is only in response to usage. Segments are identified with virtual segment identifiers that are retained when segments are moved to a different storage node. Logical volumes may therefore be moved seamlessly to different storage nodes to ensure sufficient storage capacity. Data is written to new locations in segments having space and a block map tracks the last segment to which data for a given address is written. Garbage collection is performed to free segments that contain invalid data, i.e. data for addresses that have been subsequently written to.

    Implementing Affinity And Anti-Affinity Constraints In A Bundled Application

    公开(公告)号:US20200042392A1

    公开(公告)日:2020-02-06

    申请号:US16050655

    申请日:2018-07-31

    摘要: A new snapshot of a storage volume is created by suppressing write requests. Once pending write requests from the computing nodes are completed, storage nodes create a new snapshot for the storage volume by allocating a new segment to the new snapshot. Subsequent write requests to the storage volume are then performed on the segments allocated to the new snapshot. An orchestration layer implements a bundled application that is provisioned with storage volumes and containers. A snapshot of the application may be created and used to rollback or clone the application. The amount of processing cores, memory, and containers may be increased or decreased based on usage. Components of the bundled application may be assigned to nodes to satisfy affinity and anti-affinity rules. Device sets may be generated according to these rules and used to plan for the assignment of storage volumes and containers of the bundled application.

    System and method performing job management

    公开(公告)号:US10510192B2

    公开(公告)日:2019-12-17

    申请号:US15090065

    申请日:2016-04-04

    摘要: A system provides systems and methods for visualizing a computational process. A portion of attributes of a computational process may be displayed on a face of a three-dimensional object. In response to rotation of the object by a user, a different face of the object may be displayed and a different portion of the attributes displayed thereon. Each object may represent a status of a computational cluster, specifically a set of jobs executing on the cluster. A cluster may be selected and the jobs represented by means of graphical indicators arranges on a time axis and a performance axis. The size of a graphical indicator may indicate an amount of processing time consume by the corresponding job. A color of the graphical indicator may indicate a status. A portion of the graphical indicator may be visually distinguished from a remaining portion to indicate a completion percentage of the job.

    Storage scheme for a distributed storage system

    公开(公告)号:US10423344B2

    公开(公告)日:2019-09-24

    申请号:US15709269

    申请日:2017-09-19

    IPC分类号: G06F3/06 G06F11/14

    摘要: A system maintains a consistency database that maintains a status (current, down, stale) for copies of logical storage volumes stored on storage nodes. As failures are detected, the consistency database is updated. Copies are synchronized with one another using information in the consistency database. Write operations on a primary node for a slice of a logical storage node are assigned a virtual block address (VBA) that is mapped to a logical block address (LBA) within the slice. Consistency of the VBAs of the primary node and that of a secondary node is evaluated and used to detect currency. VBA holes are detected and corresponding write commands resent to maintain currency. Physical segments on the primary node are assigned virtual segment identifiers (VSID) that are maintained consistent with VSIDs on clone nodes so that they can be used for garbage collection and synchronization.

    Upgrading Bundled Applications In A Distributed Computing System

    公开(公告)号:US20190220266A1

    公开(公告)日:2019-07-18

    申请号:US15870321

    申请日:2018-01-12

    IPC分类号: G06F8/65 G06F17/30 G06F11/14

    摘要: A new snapshot of a storage volume is created by instructing computing nodes to suppress write requests. Once pending write requests from the computing nodes are completed, storage nodes create a new snapshot for the storage volume by allocating a new segment to the new snapshot and finalizes and performs garbage collection with respect to segments allocated to the previous snapshot. Subsequent write requests to the storage volume are then performed on the segments allocated to the new snapshot. An orchestration layer implements a multi-role application that is provisioned with virtualized storage and computation resources. A snapshot of the application may be created and used to rollback or clone the application. Clones snapshots of storage volumes may be gradually populated with data from prior snapshots to reduce loading on a primary snapshot. Upgrades for bundled applications may be defined and executed for containers and roles of the bundled application.

    Implementing Secure Communication In A Distributed Computing System

    公开(公告)号:US20190215313A1

    公开(公告)日:2019-07-11

    申请号:US15868586

    申请日:2018-01-11

    IPC分类号: H04L29/06 H04L9/32 G06F11/20

    摘要: A new snapshot of a storage volume is created by instructing computing nodes to suppress write requests. Once pending write requests from the computing nodes are completed, storage nodes create a new snapshot for the storage volume by allocating a new segment to the new snapshot and finalizes and performs garbage collection with respect to segments allocated to the previous snapshot. Subsequent write requests to the storage volume are then performed on the segments allocated to the new snapshot. An orchestration layer implements a multi-role application that is provisioned with virtualized storage and computation resources. A snapshot of the application may be created and used to rollback or clone the application. Containers may be loaded with data structures for encrypted and/or authenticated communication. Failed storage volumes maybe detected and repaired. Repopulating repaired volumes may be managed by an application or a storage manager implementing a virtualized storage system.

    Implementing Fault Domain And Latency Requirements In A Virtualized Distributed Storage System

    公开(公告)号:US20190213085A1

    公开(公告)日:2019-07-11

    申请号:US15868652

    申请日:2018-01-11

    IPC分类号: G06F11/14 G06F17/30

    摘要: A new snapshot of a storage volume is created by instructing computing nodes to suppress write requests. Once pending write requests from the computing nodes are completed, storage nodes create a new snapshot for the storage volume by allocating a new segment to the new snapshot and finalizes and performs garbage collection with respect to segments allocated to the previous snapshot. Subsequent write requests to the storage volume are then performed on the segments allocated to the new snapshot. An orchestration layer implements a multi-role application that is provisioned with virtualized storage and computation resources. A snapshot of the application may be created and used to rollback or clone the application. Clones snapshots of storage volumes may be gradually populated with data from prior snapshots to reduce loading on a primary snapshot.

    Storage Scheme For A Distributed Storage System

    公开(公告)号:US20190087104A1

    公开(公告)日:2019-03-21

    申请号:US15709269

    申请日:2017-09-19

    IPC分类号: G06F3/06 G06F11/14

    摘要: A system maintains a consistency database that maintains a status (current, down, stale) for copies of logical storage volumes stored on storage nodes. As failures are detected, the consistency database is updated. Copies are synchronized with one another using information in the consistency database. Write operations on a primary node for a slice of a logical storage node are assigned a virtual block address (VBA) that is mapped to a logical block address (LBA) within the slice. Consistency of the VBAs of the primary node and that of a secondary node is evaluated and used to detect currency. VBA holes are detected and corresponding write commands resent to maintain currency. Physical segments on the primary node are assigned virtual segment identifiers (VSID) that are maintained consistent with VSIDs on clone nodes so that they can be used for garbage collection and synchronization.

    VIRTUAL NETWORK OVERLAYS FOR MULTIPATH DATACENTERS

    公开(公告)号:US20170302572A1

    公开(公告)日:2017-10-19

    申请号:US15099012

    申请日:2016-04-14

    摘要: Creation of a virtual overlay for a multipath network with tiered topology is disclosed. A virtual-node module may group network nodes connected within a common tier, which is initially the tier furthest from the core. This module may also group network nodes at a consecutive tier, closer to the core, that connect a pairs of groupings in the common tier by links between the common and consecutive tiers. Groupings with a common network node may be combined and designated as virtual nodes. A virtual-link module may sum bandwidths for network links between virtual-node pairs with nodes in both the common and consecutive tiers. A check module may combine virtual nodes and links and network nodes and links not yet virtualized in a graph, which it may check for a cycle. An iteration module may iterate both tiers toward the core, triggering further grouping, summing, and updating until no loop exists.

    Asynchronous scheduling informed by job characteristics and anticipatory provisioning of data for real-time, parallel processing
    90.
    发明授权
    Asynchronous scheduling informed by job characteristics and anticipatory provisioning of data for real-time, parallel processing 有权
    异步调度通过作业特性和预期的数据提供来实时,并行处理

    公开(公告)号:US09331943B2

    公开(公告)日:2016-05-03

    申请号:US14174473

    申请日:2014-02-06

    摘要: Systems and methods are disclosed for scheduling jobs processed in a distributed fashion to realize unharnessed efficiencies latent in the characteristics of the jobs and distributed processing technologies. A job store may be communicatively coupled to a job analyzer. The job analyzer may be operable to read information characterizing a job to identify multiple data blocks to be processed during the job at multiple locations in a cluster of nodes. A scheduling module may use information about the multiple data blocks, their storage locations, their status with respect to being provisioned to processing logic, data blocks to be processed by other jobs, data blocks in cache that have been pre-fetched for a prior job, quality-of-services parameters, and/or job characteristics, such as job size, to schedule the job in relation to other jobs.

    摘要翻译: 公开了用于以分布式方式处理的作业来实现系统和方法,以实现在作业和分布式处理技术的特征中潜在的无效率。 作业存储可以通信地耦合到作业分析器。 作业分析器可以用于读取表征作业的信息,以识别在节点簇中的多个位置处的作业期间要处理的多个数据块。 调度模块可以使用关于多个数据块的信息,它们的存储位置,它们关于被提供给处理逻辑的状态,要由其他作业处理的数据块,已被预先获取的先前作业的高速缓存中的数据块 ,服务质量参数和/或作业特征(如作业大小)来安排与其他作业相关的作业。