Abstract:
Techniques for optimizing program code through property merging are described. In an embodiment, a compiler identifies, from a plurality of properties of a particular data object that are referenced by the program code, one or more candidate sets of properties that are eligible for merging. For a respective candidate set of properties of the one or more candidate set of properties, the compiler determines whether to merge different properties of the particular data object that belong to the respective candidate set of properties. After determining to merge the different properties, a particular data structure is generated, within the memory of a computing device, that stores the different properties of the particular data object that belong to the respective candidate set.
Abstract:
Techniques herein are for navigation data structures for graph traversal. In an embodiment, navigation data structures that a computer stores include: a source vertex array of vertices; a neighbor array of dense identifiers of target vertices terminating edges; a bidirectional map associating, for each vertex, a sparse identifier of the vertex with a dense identifier of the vertex; and a vertex array containing, when a dense identifier of a source vertex is used as an offset, a pair of offsets defining an offset range, for use with the neighbor array. The source vertex array, using the dense identifier of a particular vertex as an offset, contains an offset, into a neighbor array, of a target vertex terminating an edge originating at the particular vertex. The neighbor array contiguously stores dense identifiers of target vertices terminating edges originating from a same source vertex.
Abstract:
Techniques for identifying, in a target graph, subgraphs that match a query graph are provided. Processing a query graph comprises multiple stages, one for each query node in the query graph. In the first stage, a query node is selected, different portions of the target graph are assigned to different threads, each thread identifies nodes that match the selected query node and stores the identities of those nodes in storage that is local to the thread. The results of each thread are then stored in a “global” data structure. In the second stage, a second query node is selected and different portions of the global data structure are assigned to different threads. Each thread identifies nodes that match the second query node and that are connected to a previously-matched node. The second stage repeats until all nodes in the query graph are processed.
Abstract:
An analyzer (such as a compiler) searches for a program portion that matches a pattern that may suffer from workload imbalance due to nodes with high degrees (i.e., relatively many edges). Such a pattern involves iteration over at least a subset (or all) of the nodes in a graph. If a program portion that matches the pattern is found, then the analyzer determines whether the body of the iteration contains an iteration over edges or neighbors of each node in the subset. If so, then the analyzer transforms the graph analytic program by adding code and, optionally, modifying existing code so that high-degree nodes are processed differently than low-degree nodes. High-degree nodes are processed sequentially while low-degree nodes are processed in parallel. Conversely, edges of high-degree nodes are processed in parallel while edges of low-degree nodes are processed sequentially.
Abstract:
Techniques are provided for dynamically self-balancing communication and computation. In an embodiment, each partition of application data is stored on a respective computer of a cluster. The application is divided into distributed jobs, each of which corresponds to a partition. Each distributed job is hosted on the computer that hosts the corresponding data partition. Each computer divides its distributed job into computation tasks. Each computer has a pool of threads that execute the computation tasks. During execution, one computer receives a data access request from another computer. The data access request is executed by a thread of the pool. Threads of the pool are bimodal and may be repurposed between communication and computation, depending on workload. Each computer individually detects completion of its computation tasks. Each computer informs a central computer that its distributed job has finished. The central computer detects when all distributed jobs of the application have terminated.
Abstract:
Systems and methods for interactive front-end graph analysis are provided herein. According to one embodiment, a front-end application receives, from a compiler, first meta-information for a particular graph analysis procedure, where the first meta-information identifies a set of input parameters for passing graph information to the particular graph analysis procedure. The front-end application registers, using the first meta-information, the particular graph analysis procedure as an available command. The front-end application also receives second meta-information that identifies, for each respective graph object of a set of one or more graph objects, a respective set of graph characteristics. In response to receiving a request to apply the particular graph analysis procedure to the set of one or more graph objects, the front-end application enforces a set of one or more constraints based on the first meta-information and the second meta-information.
Abstract:
Techniques for efficiently loading graph data into memory are provided. A plurality of node ID lists are retrieved from storage. Each node ID list is ordered based on one or more order criteria, such as node ID, and is read into memory. A new list of node IDs is created in memory and is initially empty. From among the plurality of node ID lists, a particular node ID is selected based on the one or more order criteria, removed from the node ID list where the particular node ID originates, and added to the new list. This process of selecting, removing, and adding continues until no more than one node ID list exists, other than the new list. In this way, the retrieval of the plurality of node ID lists from storage may be performed in parallel while the selecting and adding are performed sequentially.
Abstract:
Techniques for analyzing and modifying a graph analytic program are provided. An analyzer (such as a compiler) searches for a program portion that matches a pattern that may suffer from workload imbalance due to nodes with high degrees (i.e., relatively many edges). Such a pattern involves iteration over at least a subset (or all) of the nodes in a graph. If a program portion that matches the pattern is found, then the analyzer determines whether the body of the iteration contains an iteration over edges or neighbors of each node in the subset. If so, then the analyzer transforms the graph analytic program by adding code and, optionally, modifying existing code so that high-degree nodes are processed differently than low-degree nodes. High-degree nodes are processed sequentially while low-degree nodes are processed in parallel. Conversely, edges of high-degree nodes are processed in parallel while edges of low-degree nodes are processed sequentially.
Abstract:
Techniques for identifying, in a target graph, subgraphs that match a query graph are provided. Processing a query graph comprises multiple stages, one for each query node in the query graph. In the first stage, a query node is selected, different portions of the target graph are assigned to different threads, each thread identifies nodes that match the selected query node and stores the identities of those nodes in storage that is local to the thread. The results of each thread are then stored in a “global” data structure. In the second stage, a second query node is selected and different portions of the global data structure are assigned to different threads. Each thread identifies nodes that match the second query node and that are connected to a previously-matched node. The second stage repeats until all nodes in the query graph are processed.
Abstract:
Techniques for identifying common neighbors of two nodes in a graph are provided. One technique involves performing a binary split search and/or a linear search. Another technique involves creating a segmenting index for a first neighbor list. A second neighbor list is scanned and, for each node indicated in the second neighbor list, the segmenting index is used to determine whether the node is also indicated in the first neighbor list. Techniques are also provided for counting the number of triangles. One technique involves pruning nodes from neighbor lists based on the node values of the nodes whose neighbor lists are being pruned. Another technique involves sorting the nodes in a node array (and, thus, their respective neighbor lists) based on the nodes' respective degrees prior to identifying common neighbors. In this way, when pruning the neighbor lists, the neighbor lists of the highly connected nodes are significantly reduced.