Abstract:
Techniques for identifying, in a target graph, subgraphs that match a query graph are provided. Processing a query graph comprises multiple stages, one for each query node in the query graph. In the first stage, a query node is selected, different portions of the target graph are assigned to different threads, each thread identifies nodes that match the selected query node and stores the identities of those nodes in storage that is local to the thread. The results of each thread are then stored in a “global” data structure. In the second stage, a second query node is selected and different portions of the global data structure are assigned to different threads. Each thread identifies nodes that match the second query node and that are connected to a previously-matched node. The second stage repeats until all nodes in the query graph are processed.
Abstract:
An analyzer (such as a compiler) searches for a program portion that matches a pattern that may suffer from workload imbalance due to nodes with high degrees (i.e., relatively many edges). Such a pattern involves iteration over at least a subset (or all) of the nodes in a graph. If a program portion that matches the pattern is found, then the analyzer determines whether the body of the iteration contains an iteration over edges or neighbors of each node in the subset. If so, then the analyzer transforms the graph analytic program by adding code and, optionally, modifying existing code so that high-degree nodes are processed differently than low-degree nodes. High-degree nodes are processed sequentially while low-degree nodes are processed in parallel. Conversely, edges of high-degree nodes are processed in parallel while edges of low-degree nodes are processed sequentially.
Abstract:
Techniques are provided for dynamically self-balancing communication and computation. In an embodiment, each partition of application data is stored on a respective computer of a cluster. The application is divided into distributed jobs, each of which corresponds to a partition. Each distributed job is hosted on the computer that hosts the corresponding data partition. Each computer divides its distributed job into computation tasks. Each computer has a pool of threads that execute the computation tasks. During execution, one computer receives a data access request from another computer. The data access request is executed by a thread of the pool. Threads of the pool are bimodal and may be repurposed between communication and computation, depending on workload. Each computer individually detects completion of its computation tasks. Each computer informs a central computer that its distributed job has finished. The central computer detects when all distributed jobs of the application have terminated.
Abstract:
Systems and methods for interactive front-end graph analysis are provided herein. According to one embodiment, a front-end application receives, from a compiler, first meta-information for a particular graph analysis procedure, where the first meta-information identifies a set of input parameters for passing graph information to the particular graph analysis procedure. The front-end application registers, using the first meta-information, the particular graph analysis procedure as an available command. The front-end application also receives second meta-information that identifies, for each respective graph object of a set of one or more graph objects, a respective set of graph characteristics. In response to receiving a request to apply the particular graph analysis procedure to the set of one or more graph objects, the front-end application enforces a set of one or more constraints based on the first meta-information and the second meta-information.
Abstract:
Techniques for efficiently loading graph data into memory are provided. A plurality of node ID lists are retrieved from storage. Each node ID list is ordered based on one or more order criteria, such as node ID, and is read into memory. A new list of node IDs is created in memory and is initially empty. From among the plurality of node ID lists, a particular node ID is selected based on the one or more order criteria, removed from the node ID list where the particular node ID originates, and added to the new list. This process of selecting, removing, and adding continues until no more than one node ID list exists, other than the new list. In this way, the retrieval of the plurality of node ID lists from storage may be performed in parallel while the selecting and adding are performed sequentially.
Abstract:
Techniques for analyzing and modifying a graph analytic program are provided. An analyzer (such as a compiler) searches for a program portion that matches a pattern that may suffer from workload imbalance due to nodes with high degrees (i.e., relatively many edges). Such a pattern involves iteration over at least a subset (or all) of the nodes in a graph. If a program portion that matches the pattern is found, then the analyzer determines whether the body of the iteration contains an iteration over edges or neighbors of each node in the subset. If so, then the analyzer transforms the graph analytic program by adding code and, optionally, modifying existing code so that high-degree nodes are processed differently than low-degree nodes. High-degree nodes are processed sequentially while low-degree nodes are processed in parallel. Conversely, edges of high-degree nodes are processed in parallel while edges of low-degree nodes are processed sequentially.
Abstract:
Techniques for identifying, in a target graph, subgraphs that match a query graph are provided. Processing a query graph comprises multiple stages, one for each query node in the query graph. In the first stage, a query node is selected, different portions of the target graph are assigned to different threads, each thread identifies nodes that match the selected query node and stores the identities of those nodes in storage that is local to the thread. The results of each thread are then stored in a “global” data structure. In the second stage, a second query node is selected and different portions of the global data structure are assigned to different threads. Each thread identifies nodes that match the second query node and that are connected to a previously-matched node. The second stage repeats until all nodes in the query graph are processed.
Abstract:
Techniques for identifying common neighbors of two nodes in a graph are provided. One technique involves performing a binary split search and/or a linear search. Another technique involves creating a segmenting index for a first neighbor list. A second neighbor list is scanned and, for each node indicated in the second neighbor list, the segmenting index is used to determine whether the node is also indicated in the first neighbor list. Techniques are also provided for counting the number of triangles. One technique involves pruning nodes from neighbor lists based on the node values of the nodes whose neighbor lists are being pruned. Another technique involves sorting the nodes in a node array (and, thus, their respective neighbor lists) based on the nodes' respective degrees prior to identifying common neighbors. In this way, when pruning the neighbor lists, the neighbor lists of the highly connected nodes are significantly reduced.
Abstract:
A method for generating a binary executable of a program so that private symbols in a module are accessible from another module. In one embodiment, the method compiles a source program to an intermediate representation and scans the representation to find the private symbols in the program's modules. It then wraps a function around each private symbol. When called, the function returns an address of the private symbol, so that the other module can access the symbol from outside the module in which the symbol is found. At run time, a call is made to obtain the address of the function, which is then executed to obtain the address of the private symbol so that the symbol can be accessed. In another embodiment, a Just-In-Time compiler executes the wrapper functions and patches the executable program with the direct address of the private symbol to avoid a call to the wrapper function.
Abstract:
A method for generating a binary executable of a program so that private symbols in a module are accessible from another module. In one embodiment, the method compiles a source program to an intermediate representation and scans the representation to find the private symbols in the program's modules. It then wraps a function around each private symbol. When called, the function returns an address of the private symbol, so that the other module can access the symbol from outside the module in which the symbol is found. At run time, a call is made to obtain the address of the function, which is then executed to obtain the address of the private symbol so that the symbol can be accessed. In another embodiment, a Just-In-Time compiler executes the wrapper functions and patches the executable program with the direct address of the private symbol to avoid a call to the wrapper function.