kedro-wings
kedro-wings copied to clipboard
Feature Request: support multiple roots
For now we either define "global" root
for all checkpoints or have to specify separate root for each type of checkpoints. This is okay but sometimes we need to transfer data between locations.
Possible scenarios:
- checkpoint some of the datasets locally
- or publish reports to the different S3 bucket
For now to support such behavior I am overriding datasets in the catalog. But I believe we can make it much better by introducing locations and tag inputs/outputs with location tags. For example:
-
01_raw/base.csv
will be check-pointed in the default root location. - while
01_raw/base.csv#reports
will be saved to thereports
location.
I am not sure whether #*
suffix is the best API though. Looks like the possible interference with dataset transcoding convention should be considered.