Snowflake Lineage from Tasks and Dynamic Tables

[marystory]

Is your feature request related to a problem? Please describe. In v1.3, Snowflake's tasks and dynamic tables are missing from the lineage metadata using metadata ingestion.

Describe the solution you'd like add snowflake lineage and object details

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context I created some simple Dynamic tables.

create or replace database demo; 
create or replace schema inventory_2; 
use schema demo.inventory_2;

create or replace table inventory_updates (product_id number, category_id number, stock number, event_time timestamp);

-- Populate first round of test data 
insert into inventory_updates values 
(1, 0, 10, current_timestamp()),
(1, 0, 5, current_timestamp() + interval '1 second'),
(2, 0, 20, current_timestamp());


-- inventory targets: thresholds for treating inventory of a product as Low
create or replace table inventory_targets (product_id number, stock_target number) ;

-- Populate first round of test data
insert into inventory_targets values
(1, 10),
(2, 10);

-- Dynamic Table showing current Enventory Levels
-- Start with downstream lag and manual refresh
create or replace dynamic table inventory target_lag=downstream warehouse=Compute_wh as
select *
from inventory_updates
qualify row_number() over (partition by product_id order by event_time desc) = 1;

-- create D T showing products currentyl below stock targets
create or replace dynamic table low_inventory_warnings target_lag=downstream warehouse=Compute_wh as
select product_id, round(stock/stock_target * 100) as stock_percent
from inventory join inventory_targets using (product_id)
where stock < stock_target;

alter dynamic table low_inventory_warnings refresh;
select * from low_inventory_warnings;

Here is the graph on snowflake:

Here is how open metadata lineage looks like:

Notice there is no lineage. Plus the Type of table should be Dynamic Table not Regular Similarly, when I create pipelines using Tasks, lineage is not extracted as part of the metadata workflow.

[harshach]

@marystory you need to deploy lineage workflow to get the lineage, did you do that? cc @ulixius9

[marystory]

I see, let me try the lineage. I'll report back

[marystory]

You are right! Thanks

[marystory]

Still it would be helpful to have some indicators about the type of table and type of target lag