-
公开(公告)号:US20240111762A1
公开(公告)日:2024-04-04
申请号:US18526666
申请日:2023-12-01
申请人: Snowflake Inc.
发明人: Subramanian Muralidhar , Benoit Dageville , Thierry Cruanes , Nileema Shingte , Saurin Shah , Torsten Grabs , Istvan Cseri
IPC分类号: G06F16/242 , G06F3/06 , G06F9/54 , G06F16/16 , G06F16/22 , G06F16/23 , G06F16/2455 , G06F16/25
CPC分类号: G06F16/2423 , G06F3/0605 , G06F3/0644 , G06F3/0653 , G06F3/067 , G06F9/542 , G06F16/164 , G06F16/2282 , G06F16/2358 , G06F16/2393 , G06F16/24557 , G06F16/256
摘要: Disclosed herein are systems and methods for efficiently querying external tables. In an embodiment, a database platform receives a query that is directed at least in part to external data in an external table stored on a data storage platform that is external to the database platform. The external table includes a plurality of partitions. The database platform identifies, from external-table metadata, a subset of the plurality of partitions of the external table as including data that potentially satisfies the query. The external-table metadata is stored by the database platform. The database platform identifies data that satisfies the query by scanning the identified subset of the partitions, and responds to the query at least in part with the identified data that satisfies the query.
-
公开(公告)号:US11507571B2
公开(公告)日:2022-11-22
申请号:US16385720
申请日:2019-04-16
申请人: Snowflake Inc.
发明人: Subramanian Muralidhar , Benoit Dageville , Thierry Cruanes , Nileema Shingte , Saurin Shah , Torsten Grabs , Istvan Cseri
IPC分类号: G06F16/23 , G06F16/248 , G06F16/2458 , G06F3/06 , G06F16/25
摘要: Systems, methods, and devices for generating a materialized view over an external table. A method includes connecting a database platform to an external table such that the database platform has read access for the external table and does not have write access for the external table. The method includes generating, by the database platform, a materialized view over the external table. The method includes receiving a notification that a modification has been made to the external table, the modification comprising one or more of an addition of a file, a deletion of a file, or an update to a file in a source location for the external table. The method includes, in response to the external table being modified, refreshing the materialized view such that the materialized view comprises an accurate representation of the external table.
-
公开(公告)号:US20210390095A1
公开(公告)日:2021-12-16
申请号:US17462435
申请日:2021-08-31
申请人: Snowflake Inc.
发明人: Subramanian Muralidhar , Benoit Dageville , Thierry Cruanes , Nileema Shingte , Saurin Shah , Torsten Grabs , Istvan Cseri
IPC分类号: G06F16/23 , G06F16/248 , G06F16/2458 , G06F3/06 , G06F16/25
摘要: The subject technology receives a notification that a modification has been made to an external table, the modification comprising inserting at least one row of new data to the external table, the at least one row corresponding to a first micro-partition that includes a first portion of data from the external table prior to the inserting. The subject technology, in response to the notification indicating the modification to the external table, generates a new micro-partition different from the first micro-partition, the new micro-partition including the inserted at least one row of new data and the first portion of data from the external table. The subject technology generates a refreshed materialized view based at least in part on the generated new micro-partition such that the refreshed materialized view comprises a representation of the external table after the modification has been made.
-
公开(公告)号:US11163757B2
公开(公告)日:2021-11-02
申请号:US17200751
申请日:2021-03-12
申请人: Snowflake Inc.
发明人: Subramanian Muralidhar , Benoit Dageville , Thierry Cruanes , Nileema Shingte , Saurin Shah , Torsten Grabs , Istvan Cseri
IPC分类号: G06F3/06 , G06F16/2453 , G06F16/25 , G06F16/23 , G06F16/242
摘要: Systems, methods, and devices for querying over an external table are disclosed. A method includes connecting a database platform to an external table such that the database platform has read access for the external table and does not have write access for the external table. The method includes receiving a query comprising a predicate, the query directed at least to data in the external table. The method includes determining, based on metadata, one or more partitions in the external table comprising data satisfying the predicate. The method includes pruning, based on the metadata, all partitions in the external table that do not comprise any data satisfying the predicate. The method includes generating a query plan comprising a plurality of discrete subtasks. The method includes assigning, based on the metadata, the plurality of discrete subtasks to one or more nodes in an execution platform.
-
公开(公告)号:US20240211491A1
公开(公告)日:2024-06-27
申请号:US18599647
申请日:2024-03-08
申请人: Snowflake Inc.
发明人: Vasile Paraschiv , Saurin Shah , Marianne Shaw , Nileema Shingte
IPC分类号: G06F16/27 , G06F9/30 , G06F16/11 , G06F16/13 , G06F16/182 , G06F16/22 , G06F16/2455 , G06F16/25 , G06F16/28
CPC分类号: G06F16/278 , G06F9/3009 , G06F16/137 , G06F16/182 , G06F16/2282 , G06F16/24554 , G06F16/283 , G06F16/116 , G06F16/254
摘要: A database export system exports data using a plurality of nodes that process the data to generate structured result files that are partitioned by an export parameter in an export request. The database export system distributes the data and merges the files to avoid small file creation and increase processing speed via parallelism. The database export system generates the result files of a specified maximum size in a final format, where the files are processed merged in a temporary file format. The parallel processing is optimized and constrained per the amount of processing nodes, available memory, requested final file sizes, and operation based ordering to complete data exports in a scalable multi-stage approach.
-
公开(公告)号:US11347775B2
公开(公告)日:2022-05-31
申请号:US17463313
申请日:2021-08-31
申请人: Snowflake Inc.
发明人: Vasile Paraschiv , Saurin Shah , Marianne Shaw , Nileema Shingte
IPC分类号: G06F16/27 , G06F16/28 , G06F9/30 , G06F16/22 , G06F16/2455
摘要: A database export system exports data using a plurality of nodes that process the data to generate structured result files that are partitioned by an export parameter in an export request. The database export system distributes the data and merges the files to avoid small file creation and increase processing speed via parallelism. The database export system generates the result files of a specified maximum size in a final format, where the files are processed merged in a temporary file format. The parallel processing is optimized and constrained per the amount of processing nodes, available memory, requested final file sizes, and operation based ordering to complete data exports in a scalable multi-stage approach.
-
公开(公告)号:US11269869B2
公开(公告)日:2022-03-08
申请号:US17498382
申请日:2021-10-11
申请人: Snowflake Inc.
发明人: Subramanian Muralidhar , Benoit Dageville , Thierry Cruanes , Nileema Shingte , Saurin Shah , Torsten Grabs , Istvan Cseri
IPC分类号: G06F16/20 , G06F16/242 , G06F3/06 , G06F16/2453 , G06F16/25 , G06F16/23
摘要: Disclosed herein are systems and methods for processing queries over external tables. In an embodiment, a database platform receives a query directed at least to data in an external table stored in a storage platform that is external to the database platform. The database platform uses metadata that summarizes the data in the external table to identify one or more partitions of the external table as potentially including data satisfying the query, and generates a query plan that includes a plurality of discrete subtasks that collectively include instructions to scan the identified one or more partitions of the external table for data satisfying the query. The database platform assigns, based on the metadata, the plurality of discrete subtasks to one or more nodes in an execution platform, and refreshes the metadata in response to a threshold number of modifications being made to the external table.
-
公开(公告)号:US20210200754A1
公开(公告)日:2021-07-01
申请号:US17200751
申请日:2021-03-12
申请人: Snowflake Inc.
发明人: Subramanian Muralidhar , Benoit Dageville , Thierry Cruanes , Nileema Shingte , Saurin Shah , Torsten Grabs , Istvan Cseri
IPC分类号: G06F16/242 , G06F3/06 , G06F16/2453 , G06F16/25 , G06F16/23
摘要: Systems, methods, and devices for querying over an external table are disclosed. A method includes connecting a database platform to an external table such that the database platform has read access for the external table and does not have write access for the external table. The method includes receiving a query comprising a predicate, the query directed at least to data in the external table. The method includes determining, based on metadata, one or more partitions in the external table comprising data satisfying the predicate. The method includes pruning, based on the metadata, all partitions in the external table that do not comprise any data satisfying the predicate. The method includes generating a query plan comprising a plurality of discrete subtasks. The method includes assigning, based on the metadata, the plurality of discrete subtasks to one or more nodes in an execution platform.
-
公开(公告)号:US20200334231A1
公开(公告)日:2020-10-22
申请号:US16775110
申请日:2020-01-28
申请人: Snowflake Inc.
发明人: Subramanian Muralidhar , Benoit Dageville , Thierry Cruanes , Nileema Shingte , Saurin Shah , Torsten Grabs , Istvan Cseri
IPC分类号: G06F16/23 , G06F16/248 , G06F16/2458 , G06F16/25 , G06F3/06
摘要: Systems, methods, and devices for generating a materialized view over an external table. A method includes connecting a database platform to an external table such that the database platform has read access for the external table and does not have write access for the external table. The method includes generating, by the database platform, a materialized view over the external table. The method includes receiving a notification that a modification has been made to the external table, the modification comprising one or more of an addition of a file, a deletion of a file, or an update to a file in a source location for the external table. The method includes, in response to the external table being modified, refreshing the materialized view such that the materialized view comprises an accurate representation of the external table.
-
公开(公告)号:US20200334230A1
公开(公告)日:2020-10-22
申请号:US16385720
申请日:2019-04-16
申请人: Snowflake Inc.
发明人: Subramanian Muralidhar , Benoit Dageville , Thierry Cruanes , Nileema Shingte , Saurin Shah , Torsten Grabs , Istvan Cseri
IPC分类号: G06F16/23 , G06F16/25 , G06F16/248 , G06F16/2458 , G06F3/06
摘要: Systems, methods, and devices for generating a materialized view over an external table. A method includes connecting a database platform to an external table such that the database platform has read access for the external table and does not have write access for the external table. The method includes generating, by the database platform, a materialized view over the external table. The method includes receiving a notification that a modification has been made to the external table, the modification comprising one or more of an addition of a file, a deletion of a file, or an update to a file in a source location for the external table. The method includes, in response to the external table being modified, refreshing the materialized view such that the materialized view comprises an accurate representation of the external table.
-
-
-
-
-
-
-
-
-