kedro
kedro copied to clipboard
Make all paths relative in `kedro catalog resolve`
Description
This bug was my bad! 😅
I'd suggested in #3561 to just remove the trimming of filepaths because we were only checking for filepath
key in the catalog config but PartitionedDataset
uses the key path
which was causing an error.
But the filepaths in context.catalog._dataset_patterns
are absolute paths (because of this https://github.com/kedro-org/kedro/blob/e8f1bfd72992336ec12591b49a5fa2654217472f/kedro/framework/context/context.py#L54) . So when running kedro catalog resolve
, the output for factory datasets had absolute paths but explicit datasets were all relative paths
Eg -
"{pattern}":
type: partitions.PartitionedDataset
path: data/01_raw
dataset: pandas.CSVDataset
metadata:
path: data/01_raw/metadata.yml # just to see what happens with nested dicts
-
kedro catalog resolve
y_train: # resolved from factory
dataset: pandas.CSVDataset
metadata:
path: /Users/ankita_katiyar/kedro_projects/space/data/01_raw/metadata.yml
path: /Users/ankita_katiyar/kedro_projects/space/data/01_raw
type: partitions.PartitionedDataset
reviews: # explicit dataset entry
filepath: data/01_raw/reviews.csv
type: pandas.CSVDataset
Development notes
Developer Certificate of Origin
We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by
line in the commit message. See our wiki for guidance.
If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.
Checklist
- [ ] Read the contributing guidelines
- [ ] Signed off each commit with a Developer Certificate of Origin (DCO)
- [ ] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
- [ ] Updated the documentation to reflect the code changes
- [ ] Added a description of this change in the
RELEASE.md
file - [ ] Added tests to cover my changes
- [ ] Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team