kedro
kedro copied to clipboard
Add method to remove runtime patterns after run
Description
After the pattern resolution logic refactoring we process all the patterns (dataset
, default
, runtime
) together.
As a result of run()
we return:
datasets that aren't in the catalog and don't match a pattern in the catalog and include MemoryDataset
https://github.com/kedro-org/kedro/blob/ba981350ad57dbcaabf5fd758a9a3d4399a91f20/kedro/runner/runner.py#L108
Before the run()
we add a runtime pattern to the catalog, so we could process all intermediate outputs as MemoryDataset
.
So currently when we do two consecutive runs https://github.com/kedro-org/kedro/issues/4235 runtime pattern {default}
added after the first run affects the next runs, so that all datasets match it and we do not return anything as a result of run()
cause we think these datasets are in the catalog.
Development notes
We think that the resolution logic is correct and all patterns should be processed together as we do now. To avoid this behaviour we added method to remove runtime patters after the run, so they only live within the run and do not affect other runs.
Developer Certificate of Origin
We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by
line in the commit message. See our wiki for guidance.
If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.
Checklist
- [ ] Read the contributing guidelines
- [ ] Signed off each commit with a Developer Certificate of Origin (DCO)
- [ ] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
- [ ] Updated the documentation to reflect the code changes
- [ ] Added a description of this change in the
RELEASE.md
file - [ ] Added tests to cover my changes
- [ ] Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team