公开(公告)号：US20240320231A1

公开(公告)日：2024-09-26

申请号：US18626007

申请日：2024-04-03

Applicant: Splunk Inc.

Inventor： Arindam Bhattacharjee ,  Sourav Pal ,  Srinivas Bobba

IPC: G06F16/2458 ,  G06F16/27

CPC classification number: G06F16/2471 ,  G06F16/278

Abstract: Systems and methods are described for distributed processing a query in a first query language utilizing a query execution engine intended for single-device execution. While distributed processing provides numerous benefits over single-device processing, distributed query execution engines can be significantly more difficult to develop that single-device engines. Embodiments of this disclosure enable the use of a single-device engine to support distributed processing, by dividing a query into multiple stages, each of which can be executed by multiple, concurrent executions of a single-device engine. Between stages, data can be shuffled between executions of the engine, such that individual executions of the engine are provided with a complete set of records needed to implement an individual stage. Because single-device engines can be significantly less difficult to develop, use of the techniques described herein can enable a distributed system to rapidly support multiple query languages.

公开(公告)号：US20240220497A1

公开(公告)日：2024-07-04

申请号：US18609798

申请日：2024-03-19

Applicant: Splunk Inc.

Inventor： Sourav Pal ,  Arindam Bhattacharjee

IPC: G06F16/2453 ,  G06F16/21 ,  G06F16/2455 ,  G06F16/2458 ,  G06F16/25 ,  G06F16/28 ,  G06F40/205

CPC classification number: G06F16/24535 ,  G06F16/219 ,  G06F16/24554 ,  G06F16/24568 ,  G06F16/2471 ,  G06F16/25 ,  G06F16/288 ,  G06F40/205

Abstract: Systems and methods are disclosed for executing a query that includes an indication to process data managed by an external data system. The system identifies the external data system that manages the data to be processed and generates a subquery for the external data system indicating that the results of the subquery are to be sent to one worker node of multiple worker nodes. The system instructs the one worker node to distribute the results received from the external data system to multiple worker nodes for processing.

公开(公告)号：US12007996B2

公开(公告)日：2024-06-11

申请号：US18051481

申请日：2022-10-31

Applicant: Splunk Inc.

Inventor： Balaji Rao ,  Jindrich Dinga ,  Kieran Cairney ,  Manuel Martinez ,  Nitilaksha Halakatti ,  Ningxuan He ,  Arindam Bhattacharjee ,  Sourav Pal ,  Alexandros Batsakis

IPC: G06F15/16 ,  G06F8/61 ,  G06F16/2453 ,  G06F16/2458 ,  H04L9/08 ,  H04L41/0806 ,  H04L67/10 ,  H04L67/52

CPC classification number: G06F16/24547 ,  G06F8/61 ,  G06F16/2465 ,  H04L9/0866 ,  H04L41/0806 ,  H04L67/10 ,  H04L67/52

Abstract: Systems and methods are described for establishing and managing components of a distributed computing framework implemented in a data intake and query system. The distributed computing framework may include a master and a plurality of worker nodes. The master may selectively operate on a search head captain that is chosen from the search heads of the data intake and query system. The search head captain may distribute configuration information for the master and the distributed computing framework to the other search heads, which in turn, may distribute that configuration information to indexers of the data intake and query system. Worker nodes may be selectively activated for operation on the indexers based on the configuration information, and the worker nodes may additionally use the configuration information to contact the master and join the distributed computing framework. This approach may provide numerous benefits, including improved security, flexibility in the selection of worker nodes, and redundancy for failures of physical components of the data intake and query system.

公开(公告)号：US11966391B2

公开(公告)日：2024-04-23

申请号：US18162646

申请日：2023-01-31

Applicant: Splunk Inc.

Inventor： Sourav Pal ,  Arindam Bhattacharjee

IPC: G06F17/00 ,  G06F16/21 ,  G06F16/2453 ,  G06F16/2455 ,  G06F16/2458 ,  G06F16/25 ,  G06F16/28 ,  G06F40/205

CPC classification number: G06F16/24535 ,  G06F16/219 ,  G06F16/24554 ,  G06F16/24568 ,  G06F16/2471 ,  G06F16/25 ,  G06F16/288 ,  G06F40/205

Abstract: Systems and methods are disclosed for executing a query that includes an indication to process data managed by an external data system. The system identifies the external data system that manages the data to be processed and generates a subquery for the external data system indicating that the results of the subquery are to be sent to one worker node of multiple worker nodes. The system instructs the one worker node to distribute the results received from the external data system to multiple worker nodes for processing.

公开(公告)号：US11860940B1

公开(公告)日：2024-01-02

申请号：US17233193

申请日：2021-04-16

Applicant: Splunk Inc.

Inventor： Alexandros Batsakis ,  Ashish Mathew ,  Christopher Madden Pride ,  Bharath Kishore Reddy Aleti ,  Sourav Pal ,  Arindam Bhattacharjee ,  James Monschke

IPC: G06F16/901 ,  G06F16/903 ,  G06F16/2458

CPC classification number: G06F16/901 ,  G06F16/2477 ,  G06F16/90335

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system uses a search node catalog to identify search nodes that are available to execute the query and uses a bucket catalog to identify buckets to be searched. The data intake and query system executes the query using the identified bucket and search nodes.

公开(公告)号：US20230214386A1

公开(公告)日：2023-07-06

申请号：US18181900

申请日：2023-03-10

Applicant: Splunk Inc.

Inventor： Sourav Pal ,  Arindam Bhattacharjee

IPC: G06F16/2453 ,  G06F16/242 ,  G06F16/25 ,  G06F16/22

CPC classification number: G06F16/24535 ,  G06F16/2425 ,  G06F16/258 ,  G06F16/22

Abstract: Systems and methods are disclosed for receiving, at a data intake and query system, a query that includes an indication to process data managed by a third-party data storage and processing system that supports a different query language than the data intake and query system. The data intake and query system identifies a third-party data storage and processing system that manages the data to be processed and generates a subquery for execution by the third-party data storage and processing system, generates instructions for one or more worker nodes to receive and process results of the subquery from the third-party data storage and processing system, and instructs the worker nodes to provide results of the processing to the data intake and query system.

公开(公告)号：US20230177047A1

公开(公告)日：2023-06-08

申请号：US18162646

申请日：2023-01-31

Applicant: Splunk Inc.

Inventor： Sourav Pal ,  Arindam Bhattacharjee

IPC: G06F16/2453 ,  G06F16/25 ,  G06F16/21 ,  G06F16/28 ,  G06F16/2455 ,  G06F16/2458 ,  G06F40/205

CPC classification number: G06F16/24535 ,  G06F16/25 ,  G06F16/219 ,  G06F16/288 ,  G06F16/24554 ,  G06F16/24568 ,  G06F16/2471 ,  G06F40/205

Abstract: Systems and methods are disclosed for executing a query that includes an indication to process data managed by an external data system. The system identifies the external data system that manages the data to be processed and generates a subquery for the external data system indicating that the results of the subquery are to be sent to one worker node of multiple worker nodes. The system instructs the one worker node to distribute the results received from the external data system to multiple worker nodes for processing.

公开(公告)号：US20230144450A1

公开(公告)日：2023-05-11

申请号：US18051470

申请日：2022-10-31

Applicant: Splunk Inc.

Inventor： Arindam Bhattacharjee ,  Sourav Pal ,  Christopher Pride

IPC: G06F16/2455 ,  G06F11/30 ,  G06F7/53 ,  G06F16/27 ,  G06F11/34

CPC classification number: G06F16/24554 ,  G06F7/5324 ,  G06F11/3006 ,  G06F11/3086 ,  G06F11/3433 ,  G06F16/278 ,  G06F2201/86 ,  G06F2201/835

Abstract: Systems and methods are disclosed for processing and executing queries against one or more dataset. As part of processing the query, the system determines whether the query is susceptible to a significantly imbalanced partition. In the event, the query is susceptible to an imbalanced partition, the system monitors the query and determines whether to perform a multi-partitioning determination to avoid a significantly imbalanced partition.

公开(公告)号：US11580107B2

公开(公告)日：2023-02-14

申请号：US16398038

申请日：2019-04-29

Applicant: Splunk Inc.

Inventor： Sourav Pal ,  Arindam Bhattacharjee ,  Asha Andrade ,  Nikhil Roy

IPC: G06F16/00 ,  G06F16/2455 ,  G06F9/50 ,  G06F16/22

Abstract: Systems and methods are described for exporting bucket data from one or more buckets to one or more worker nodes. The system can identify data from different bucket data from buckets stored in a data intake and query system that is to be processed by one or more worker nodes. The system can allocate one or more execution resources, such as a processing pipeline, to process and export the bucket data from the buckets. The system can assign bucket data corresponding to individual buckets to the execution resource based on a bucket distribution policy. The indexer can export the bucket data to the worker nodes for further processing based on the bucket data-execution resource assignment.

公开(公告)号：US11442935B2

公开(公告)日：2022-09-13

申请号：US16397930

申请日：2019-04-29

Applicant: Splunk Inc.

Inventor： Sourav Pal ,  Arindam Bhattacharjee ,  Asha Andrade

IPC: G06F16/00 ,  G06F16/2453 ,  G06F16/2455 ,  G06F9/50 ,  G06F16/2458

Abstract: Systems and methods are described for determining a record generation estimate related to a particular processing task. The system obtains a sample set of data that includes multiple records. The system applies a processing task, such as a transform or regular expression rule to the sample set of data and determines how many records are generated by the processing task. Based on the number of records generated, the system determines a record generation estimate. The system can use the record generation estimate to allocate compute resources or determine a query execution time for at least a portion of the query based on the record generation estimate.