-
公开(公告)号：US20210311957A1
公开(公告)日：2021-10-07
申请号：US16842243
申请日：2020-04-07
Applicant: Snowflake Inc.
Inventor： Gjorgji Georgievski , Ganeshan Ramachandran Iyer , Dinesh Chandrakant Kulkarni , Jiaxing Liang , Subramanian Muralidhar
Abstract: Embodiments of the present disclosure may provide cross cloud auto-ingestion techniques. A deployment may monitor multiple queues across different cloud providers and may classify the queues based on their cloud. provider type. The deployment may receive notifications from those queues regarding new data ready for ingestion. The deployment may maintain a pool of credentials and may assign appropriate credentials to each queue. The deployment may route the notifications to appropriate receivers based on cloud provider types. The receivers may then auto-ingest new data in the corresponding queue.
-
公开(公告)号：US20250068622A1
公开(公告)日：2025-02-27
申请号：US18948663
申请日：2024-11-15
Applicant: Snowflake Inc.
Inventor： Abdullah Al Mahmood , Tyler Jones , Xin Huang , Ganeshan Ramachandran Iyer , Jiaxing Liang , Daniel Mills , Subramanian Muralidhar , Daniel E. Sotolongo
Abstract: Techniques for creating and using error tables to track errors associated with a base table are described. A command to perform an operation on a base table stored in a network-based data system can be received and executed, causing at least one error. At least one error record corresponding to the at least one error can be inputted into an error table, which is nested with the base table. Contextual information can be added to the at least one error record.
-
公开(公告)号：US20240386010A1
公开(公告)日：2024-11-21
申请号：US18426772
申请日：2024-01-30
Applicant: Snowflake Inc.
Inventor： Abdullah Al Mahmood , Tyler Jones , Xin Huang , Ganeshan Ramachandran Iyer , Jiaxing Liang , Daniel Mills , Subramanian Muralidhar , Daniel E. Sotolongo
Abstract: Techniques for creating and using error tables to track errors associated with a base table are described. A command to perform an operation on a base table stored in a network-based data system can be received and executed, causing at least one error. At least one error record corresponding to the at least one error can be inputted into an error table, which is nested with the base table. Contextual information can be added to the at least one error record.
-
公开(公告)号：US20230376391A1
公开(公告)日：2023-11-23
申请号：US18158586
申请日：2023-01-24
Applicant: Snowflake Inc.
Inventor： Abdullah Al Mahmood , Ruta Dhaneshwar , Xin Huang , Ganeshan Ramachandran Iyer , Jiaxing Liang , Nithin Mahesh , Raghav Ramachandran , Purav B. Saraiya , Yanyi Zhang
CPC classification number: G06F11/2028 , G06F16/27 , G06F11/1435
Abstract: Described herein are techniques for improving disaster recovery, in particular disaster recovery pertaining to data transfer requests. The data transfer request can be received by each of multiple deployments; however, only a primary deployment can process the request. The data transferred by the primary deployment may be replicated in the secondary deployments. In response to a failover event, one of the secondary deployments can be designated as the new primary development and continue the data transfer based on the data transfer request and the replication information received from the old primary deployment prior to the failover.
-
公开(公告)号：US11586515B1
公开(公告)日：2023-02-21
申请号：US17663941
申请日：2022-05-18
Applicant: Snowflake Inc.
Inventor： Abdullah Al Mahmood , Ruta Dhaneshwar , Xin Huang , Ganeshan Ramachandran Iyer , Jiaxing Liang , Nithin Mahesh , Raghav Ramachandran , Purav B. Saraiya , Yanyi Zhang
Abstract: Described herein are techniques for improving disaster recovery, in particular disaster recovery pertaining to data transfer requests. The data transfer request can be received by each of multiple deployments; however, only a primary deployment can process the request. The data transferred by the primary deployment may be replicated in the secondary deployments. In response to a failover event, one of the secondary deployments can be designated as the new primary development and continue the data transfer based on the data transfer request and the replication information received from the old primary deployment prior to the failover.
-
公开(公告)号：US11294890B2
公开(公告)日：2022-04-05
申请号：US16365219
申请日：2019-03-26
Applicant: Snowflake Inc.
Inventor： Jiansheng Huang , Jiaxing Liang , Scott Ziegler , Haowei Yu , Benoit Dageville , Varun Ganesh
Abstract: Systems, methods, and devices for batch ingestion of data into a table of a database. A method includes determining a notification indicating a presence of a user file received from a client account to be ingested into a database. The method includes identifying data in the user file and identifying a target table of the database to receive the data in the user file. The method includes generating an ingest task indicating the data and the target table. The method includes assigning the ingest task to an execution node of an execution platform, wherein the execution platform comprises a plurality of execution nodes operating independent of a plurality of shared storage devices collectively storing database data. The method includes registering metadata concerning the target table in a metadata store after the data has been fully committed to the target table by the execution node.
-
公开(公告)号：US10997163B2
公开(公告)日：2021-05-04
申请号：US16943251
申请日：2020-07-30
Applicant: Snowflake Inc.
Inventor： Benoit Dageville , Varun Ganesh , Jiansheng Huang , Jiaxing Liang , Haowei Yu , Scott Ziegler
Abstract: The subject technology at a data system, an ingest request to ingest one or more files into a table. The subject technology, after obtaining the ingest request and prior to the ingesting of the one or more files, persists the one or more files in a first file queue that corresponds to the table, the first file queue further corresponding to a client account, and the data system further comprising a second file queue that corresponds to both a second client account and a second table. The subject technology ingests, by one or more execution nodes, the one or more files into one or more micro-partitions of the table, each of the one or more micro-partitions comprising contiguous units of storage of a storage device.
-
公开(公告)号：US10977245B2
公开(公告)日：2021-04-13
申请号：US16942421
申请日：2020-07-29
Applicant: Snowflake Inc.
Inventor： Benoit Dageville , Varun Ganesh , Jiansheng Huang , Jiaxing Liang , Haowei Yu , Scott Ziegler
Abstract: The subject technology obtains, at a database system, an ingest request to ingest one or more files into a table of a database. The subject technology, after obtaining the ingest request and prior to the ingesting of the one or more files, persists the one or more files in a file queue that corresponds to the table. The subject technology assigns the one or more files to one or more execution nodes to be ingested into the table. The subject technology operates an ingest puller to poll the file queue. The subject technology ingests, by the one or more execution nodes, the one or more files into one or more micro-partitions of the table via one or more pipes.
-
公开(公告)号：US20240411651A1
公开(公告)日：2024-12-12
申请号：US18810853
申请日：2024-08-21
Applicant: Snowflake Inc.
Inventor： Abdullah Al Mahmood , Ruta Dhaneshwar , Xin Huang , Ganeshan Ramachandran Iyer , Jiaxing Liang , Nithin Mahesh , Raghav Ramachandran , Purav B. Saraiya , Yanyi Zhang
Abstract: Described herein are techniques for improving disaster recovery, in particular disaster recovery pertaining to data transfer requests. The data transfer request can be received by each of multiple deployments; however, only a primary deployment can process the request. The data transferred by the primary deployment may be replicated in the secondary deployments. In response to a failover event, one of the secondary deployments can be designated as the new primary development and continue the data transfer based on the data transfer request and the replication information received from the old primary deployment prior to the failover.
-
公开(公告)号：US11983165B1
公开(公告)日：2024-05-14
申请号：US18128212
申请日：2023-03-29
Applicant: Snowflake Inc.
Inventor： Abdullah Al Mahmood , Chong Han , Ganeshan Ramachandran Iyer , Jiaxing Liang , Nithin Mahesh , Yanrui Zhang
IPC: G06F16/23 , G06F16/174 , G06F16/27
CPC classification number: G06F16/2365 , G06F16/1748 , G06F16/27
Abstract: Embodiments of the present disclosure provide techniques for deduplicating files during internal stage replication using a directory table of the replicated internal stage that is modified as a cache for storing and retrieving original file-level metadata for the replicated files. An initial list of candidate files for loading from the internal stage to a table of the target deployment is prepared based on the files listed in the internal stage, and refined using a directory table lookup. If there is any inconsistency between the files registered in the directory table and the files listed in the internal stage, the target deployment will inspect the user-defined file-level metadata to obtain original file-level metadata for each file that is present in the internal stage but not in the directory table. This information may be used during deduplication to ensure that no duplicate files are loaded.
