kedro-plugins icon indicating copy to clipboard operation
kedro-plugins copied to clipboard

`kedro-airflow`: Grouping in memory is not working properly

Open ankatiyar opened this issue 10 months ago • 1 comments

Description

The --group-in-memory flag is a new addition https://github.com/kedro-org/kedro-plugins/pull/241 and hasn't been released yet.

Running kedro airflow create --group-in-memory and kedro airflow create with the spaceflights project creates the exact same DAGs - there's some issue in the determining whether a dataset is MemoryDataset

https://github.com/kedro-org/kedro-plugins/blob/fbe545f63fe84b1c3610c091711429f02c7416e4/kedro-airflow/kedro_airflow/grouping.py#L14-L15

^ returns None for datasets that are not in the catalog (dataset factory datasets or MemoryDatasets

There might also be some problems with dataset factories - still investigating

ankatiyar avatar Apr 16 '24 15:04 ankatiyar

cc @sbrugman FYI!

astrojuanlu avatar Apr 16 '24 15:04 astrojuanlu