Snowflake metadata collector
The Snowflake metadata collector provides read-only access to metadata from an external Snowflake account.
The collector harvests metadata from Snowflake Data Cloud. It collects tabular objects, Horizon objects such as policies and tags, and Snowflake data quality artifacts. It also harvests functions and stored procedures. The collector can gather column statistics and table query counts. In addition, it can harvest metadata for Streamlit applications.
Metadata cataloged
The Snowflake collector catalogs the following information.
| Object | Information cataloged |
|---|---|
| Column | Name, Comments, Data Type, Is Nullable, Default Value, Key type (Primary, foreign) |
| Table | Name, Comments, Primary key, Schema, External URL (Snowsight) Extended metadata: Is Iceberg, Is dynamic, Is transient, Created Date, Last Modified Date, Last Updated By (Snowflake LAST_DDL_BY), Owner (Snowflake role that owns table), Definition Altered Date, Table Size |
| View | Name, Comments, Schema, External URL (Snowsight), Definition Altered Date, Last Modified By (Snowflake LAST_DDL_BY), Is Iceberg, Is dynamic, Is temporary, Created Date, Last Modified Date, Owner (Snowflake role that owns table), SQL definition |
| Materialized View | Name, Comments, Schema, External URL (Snowsight), Definition Altered Date, Last Modified By (Snowflake LAST_DDL_BY), Is Iceberg, Is dynamic, Is temporary, Created Date, Last Modified Date, Owner (Snowflake role that owns table) |
| Schema | Identifier, Name Extended metadata: Comments, Created date, Updated |
| Database | Type, Name, Identifier, Server, Port, Environment, JDBC URL Extended metadata: Comments |
| User-defined function (this includes User-defined functions created in Snowpark) |
Name Extended metadata: Function Definition, Comments, Created Date, Owner (Snowflake Role), Snowflake Function Type (Return Type), Function Signature, Function Language, Packages, Runtime version, Updated |
| Stored Procedure (this includes Stored Procedures created in Snowpark) |
Name Extended metadata: Function Definition, Owner (Snowflake Role), Comments, Return Type, Function Language, Packages, Created Date, Function signature, Snowflake Function Type, Runtime Version, Updated, Stored Procedure Type |
Relationships between objects
The harvested metadata includes catalog pages for the following data asset types. Each catalog page has a relationship to the other related data asset types.
| Data asset page | Relationships |
|---|---|
| Snowflake Row access policies | Tables |
| Snowflake Masking Policies | Columns, Tag value |
| Table | Columns, Tag value, Data Metric Function that analyzes, Views, Tables, or Functions which the Table depends on |
| Snowflake Tags | Tag value |
| Columns | Table, Tag Value, System Tag Value |
| View | Schema that contains Views, Columns that are part of Views, Data Metric Function that analyzes View, Views, Tables, or Functions which the View depends on |
| Materialized View | Schema that contains Materialized Views, Columns that are part of Materialized Views |
| Schema | Database that contains Schema, Table that is part of Schema, View that is part of Schema, Materialized View that is part of Schema, Streamlit app that is part of schema |
| Streamlit App | Schema that contains Streamlit App |
| Database | Schema contained in Database |
| Data Metric Function | Table or View analyzed by this Snowflake Data Metric Function, Table, Views, Tables, or Functions which the Function depends on |
Lineage and dependencies for Snowflake
The following lineage information is collected by the Snowflake collector. Note that any lineage for SQL Statements defined via variable statements are not supported.
- Any lineage for SQL Statements defined via variable statements are not supported.
- The collector traces these relationships from a View’s columns to ultimate source Table columns across SQL expressions and subqueries.
| Object | Lineage available |
|---|---|
| Column in View |
The collector identifies the associated column in an upstream view or table:
|
| User-defined function |
A Function and:
A View and Function referencing View |
| Stored Procedure |
The collector identifies:
Note: The following stored procedures are not supported:
|
Dependencies
The Snowflake collector catalogs dependencies between tables, views, and functions using the Snowflake Account Usage view OBJECT_DEPENDENCIES. See the Snowflake documentation for more information about the OBJECT_DEPENDENCIES view.
If you include the profiling and sampling specific parameters while running the collector, the following additional information is harvested for Columns.
| Object | Information cataloged |
|---|---|
| Column |
|
| Table |
|
Following additional information is cataloged when you run the collector with the Collect Snowflake policy information and Collect Snowflake tag information options enabled.
| Object | Information cataloged |
|---|---|
| Columns | Snowflake masking policies |
| Snowflake Tags | Name, Allowed Values, Comments |
| Snowflake Tag Values | Name, Value |
| Snowflake Masking Policies | Name, Comments , Date created, Snowflake tags, Owner, Type, Policy body, Schema, Database |
| Snowflake Row access policies | Name, Comments , Date created, Snowflake tags, Owner, Type, Policy body, Schema, Database |
Following additional information is cataloged when you run the collector with the Collect Snowflake table usage information and Table usage lookback days options enabled.
| Object | Information cataloged |
|---|---|
| Table | Query Count |
Following additional information about data quality metrics is cataloged when you run the collector with the Collect data metric function information option enabled.
| Object | Information cataloged |
|---|---|
| Data Metric Function |
|
| Table |
|
Following additional information is cataloged when you run the collector with theCollect Streamlit app information option enabled.
| Object | Information cataloged |
|---|---|
| Snowflake Streamlit App |
|