kedro-plugins
kedro-plugins copied to clipboard
`kedro-airflow`: Grouping in memory is not working properly
Description
The --group-in-memory
flag is a new addition https://github.com/kedro-org/kedro-plugins/pull/241 and hasn't been released yet.
Running kedro airflow create --group-in-memory
and kedro airflow create
with the spaceflights project creates the exact same DAGs - there's some issue in the determining whether a dataset is MemoryDataset
https://github.com/kedro-org/kedro-plugins/blob/fbe545f63fe84b1c3610c091711429f02c7416e4/kedro-airflow/kedro_airflow/grouping.py#L14-L15
^ returns None
for datasets that are not in the catalog (dataset factory datasets or MemoryDatasets
There might also be some problems with dataset factories - still investigating
cc @sbrugman FYI!