Invention Application
- Patent Title: ADDRESSING MEMORY LIMITS FOR PARTITION TRACKING AMONG WORKER NODES
-
Application No.: US16657867Application Date: 2019-10-18
-
Publication No.: US20200065303A1Publication Date: 2020-02-27
- Inventor: Arindam Bhattacharjee , Sourav Pal , Srinivas Bobba
- Applicant: Splunk Inc.
- Main IPC: G06F16/2458
- IPC: G06F16/2458 ; G06F16/242 ; G06F16/27 ; G06F11/07 ; G06F11/30 ; G06F9/50

Abstract:
Systems and methods are described for distributed processing a query in a first query language utilizing a query execution engine intended for single-device execution. While distributed processing provides numerous benefits over single-device processing, distributed query execution engines can be significantly more difficult to develop that single-device engines. Embodiments of this disclosure enable the use of a single-device engine to support distributed processing, by dividing a query into multiple stages, each of which can be executed by multiple, concurrent executions of a single-device engine. Between stages, data can be shuffled between executions of the engine, such that individual executions of the engine are provided with a complete set of records needed to implement an individual stage. Because single-device engines can be significantly less difficult to develop, use of the techniques described herein can enable a distributed system to rapidly support multiple query languages.
Public/Granted literature
- US11989194B2 Addressing memory limits for partition tracking among worker nodes Public/Granted day:2024-05-21
Information query