soorgeon
soorgeon copied to clipboard
soorgeon adding a section prefix to each task
Ideally when converting this pipeline the section (which is repeating therefore redundant) shouldn't appear there, or at least the prefix can be parameterized.
- source: tasks/section-1-load-data.py
product:
y: output/section-1-load-data-y.pkl
nb: output/section-1-load-data.ipynb
- source: tasks/section-2-preprocess-data.py
product:
y: output/section-2-preprocess-data-y.pkl
nb: output/section-2-preprocess-data.ipynb
- source: tasks/section-3-visualize-data.py
product:
nb: output/section-3-visualize-data.ipynb```
yeah, I did that as an easy way to ensure that the products and tasks are unique, since ploomber does not allow duplicates. but it'd be better to only add the prefix when needed (e.g. when there are two sections or products with the same name)