Invention Publication
- Patent Title: ADDRESSING MEMORY LIMITS FOR PARTITION TRACKING AMONG WORKER NODES
-
Application No.: US18626007Application Date: 2024-04-03
-
Publication No.: US20240320231A1Publication Date: 2024-09-26
- Inventor: Arindam Bhattacharjee , Sourav Pal , Srinivas Bobba
- Applicant: Splunk Inc.
- Applicant Address: US CA San Francisco
- Assignee: Splunk Inc.
- Current Assignee: Splunk Inc.
- Current Assignee Address: US CA San Francisco
- Main IPC: G06F16/2458
- IPC: G06F16/2458 ; G06F16/27

Abstract:
Systems and methods are described for distributed processing a query in a first query language utilizing a query execution engine intended for single-device execution. While distributed processing provides numerous benefits over single-device processing, distributed query execution engines can be significantly more difficult to develop that single-device engines. Embodiments of this disclosure enable the use of a single-device engine to support distributed processing, by dividing a query into multiple stages, each of which can be executed by multiple, concurrent executions of a single-device engine. Between stages, data can be shuffled between executions of the engine, such that individual executions of the engine are provided with a complete set of records needed to implement an individual stage. Because single-device engines can be significantly less difficult to develop, use of the techniques described herein can enable a distributed system to rapidly support multiple query languages.
Information query