Abstract:
Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets using containerized indexing nodes instantiated in a containerized environment. The data intake and query system stores the buckets in a shared storage system.
Abstract:
Multi-threaded processing of search responses returned by search peers is disclosed. An example method may include transmitting, by a computer system, a search request to a plurality of search peers of a data aggregation and analysis system; receiving, by a first processing thread, a plurality of data packets from the plurality of search peers; parsing, by a second processing thread, one or more data packets of the plurality of data packets to produce a first partial response to the search request; parsing, by a third processing thread, the one or more data packets to produce a second partial response to the search request; and generating, based on the first partial response and the second partial response, an aggregated response to the search request.
Abstract:
Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system identifies buckets that are to be searched and stores a copy of buckets in memory associated with one or more search nodes. A search node performs a search on buckets residing in its memory.
Abstract:
Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system parses the query and uses a metadata catalog to dynamically identify configuration parameters of datasets and/or rules associated with the query. The identified configuration parameters are communicated to a query processing component of the data intake and query system for use in executing the query.
Abstract:
Multi-threaded processing of search responses returned by search peers is disclosed. An example method may include transmitting, by a computer system, a search request to a plurality of search peers of a data aggregation and analysis system; receiving, by a first processing thread, a plurality of data packets from the plurality of search peers; parsing, by a second processing thread operating asynchronously with respect to the first processing thread, one or more data packets of the plurality of data packets, to produce a partial response to the search request; splitting the partial response into two or more fields; and generating, based on the two or more fields of the partial response, an aggregated response to the search request.
Abstract:
Processing of search responses returned by search peers is disclosed. An example method may include transmitting, by a computer system, a search request to a plurality of search peers of a data aggregation and analysis system; receiving a plurality of data packets from the plurality of search peers; parsing one or more data packets of the plurality of data packets, to produce a response to the search request; and splitting the response into two or more fields based on at least one of: a defined set of bit positions or a defined separator.
Abstract:
Processing of search responses returned by search peers is disclosed. An example method may include transmitting, by a computer system, a search request to a plurality of search peers of a data aggregation and analysis system; receiving a plurality of data packets from the plurality of search peers; parsing one or more data packets of the plurality of data packets, to produce a response to the search request; and splitting the response into two or more fields based on at least one of: a defined set of bit positions or a defined separator.
Abstract:
Multi-thread processing of search responses is disclosed. An example method may include transmitting, by a computer system, a search request to a plurality of search peers of a data aggregation and analysis system; receiving a plurality of data packets from the plurality of search peers; parsing, by a first processing thread of the computer system, one or more data packets of the plurality of data packets, to produce a partial response to the search request; and processing, by a second processing thread of the computer system, the partial response to produce a memory data structure representing an aggregated response to the search request.
Abstract:
Asynchronous processing of messages that are received from multiple servers is disclosed. An example method may include transmitting, by a computer system, a search request to a plurality of search peers of a data aggregation and analysis system. The method may further include receiving a plurality of sub-application layer protocol packets from the plurality of search peers. The method may further include parsing, by a first processing thread of the computer system, one or more sub-application layer protocol packets of the plurality of sub-application layer protocol packets, to produce an application layer message representing a partial response to the search request. The method may further include processing, by a second processing thread of the computer system, the application layer message to produce a memory data structure representing an aggregated response to the search request.
Abstract:
Systems and methods for asynchronous processing of messages that are received from multiple servers. An example method may comprise: receiving, by a first processing thread, in a non-blocking mode, a plurality of sub-application layer protocol packets from a plurality of servers; processing one or more sub-application layer protocol packets received from a first server of the plurality of servers, to produce a first application layer message; writing the first application layer message to a message queue; processing one or more sub-application layer protocol packets received from a second server of the plurality of servers, to produce a second application layer message; writing the second application layer message to the message queue; and reading, by two or more processing threads of a processing thread pool, two or more application layer messages including the first application layer message and the second application layer message from the message queue, to produce two or more memory data structures based on the read application layer messages.