DYNAMICALLY EXECUTING DATA SOURCE AGNOSTIC DATA PIPELINE CONFIGURATIONS

    公开(公告)号:US20240168800A1

    公开(公告)日:2024-05-23

    申请号:US18057874

    申请日:2022-11-22

    摘要: The disclosure describes embodiments of systems, methods, and non-transitory computer readable storage media that dynamically execute data source agnostic data pipeline job configurations that can interact with a variety of data sources while utilizing a unified request format. In particular, the disclosed systems can facilitate a data pipeline framework that utilizes source connectors for data sources, target connectors for data sources, and data transformations in data pipeline job configurations to build various data pipelines. For instance, the disclosed systems can utilize a data pipeline job configuration that includes requests for a data source in a given language with various other data pipeline functionalities via data source connectors specified within the data pipeline job configuration. For example, the disclosed systems can utilize a data source connector to map data source requests to native code commands for the data source to read or write data in relation to the data source.