kedro-wings icon indicating copy to clipboard operation
kedro-wings copied to clipboard

Feature Request: support multiple roots

Open Sitin opened this issue 3 years ago • 0 comments

For now we either define "global" root for all checkpoints or have to specify separate root for each type of checkpoints. This is okay but sometimes we need to transfer data between locations.

Possible scenarios:

  • checkpoint some of the datasets locally
  • or publish reports to the different S3 bucket

For now to support such behavior I am overriding datasets in the catalog. But I believe we can make it much better by introducing locations and tag inputs/outputs with location tags. For example:

  • 01_raw/base.csv will be check-pointed in the default root location.
  • while 01_raw/base.csv#reports will be saved to the reports location.

I am not sure whether #* suffix is the best API though. Looks like the possible interference with dataset transcoding convention should be considered.

Sitin avatar Dec 18 '20 11:12 Sitin