astro-sdk
astro-sdk copied to clipboard
Create `upstream_tasks` parameter for dependencies independent of data transfers
Description
What is the current behavior?
Currently there is "native" way to set dependencies between tasks without passing either a table or a dataframe between said tasks. This is especially apparent when chaining raw_sql functions such as here:
https://github.com/astronomer/astro-sdk/issues/524
In order to have a solution that works we need somehting that feels pythonic and doesn't rely on traditional airflow >> operators
closes: #524
What is the new behavior?
This PR adds in the ability to set dependencies using a parameter upstream_tasks. This parameter can be used at runtime to set deps without passing data. Notably we don't want to actually render the data (imagine a use-case where a task relies on 10 dataframe tasks)
Does this introduce a breaking change?
No
Checklist
- [ ] Created tests which fail without the change (if possible)
- [ ] Extended the README / documentation, if necessary
- [ ] Add documentation around
upstream_tasks - [ ] solution should not render the data since the data is never used.
Codecov Report
Merging #585 (a5a6e9f) into main (62303bd) will decrease coverage by
0.13%. The diff coverage is88.57%.
@@ Coverage Diff @@
## main #585 +/- ##
==========================================
- Coverage 93.26% 93.12% -0.14%
==========================================
Files 43 45 +2
Lines 1855 1877 +22
Branches 232 237 +5
==========================================
+ Hits 1730 1748 +18
- Misses 97 100 +3
- Partials 28 29 +1
| Impacted Files | Coverage Δ | |
|---|---|---|
| ...sdk/src/astro/sql/operators/upstream_task_mixin.py | 69.23% <69.23%> (ø) |
|
| python-sdk/src/astro/sql/operators/append.py | 100.00% <100.00%> (ø) |
|
| ...thon-sdk/src/astro/sql/operators/base_decorator.py | 93.13% <100.00%> (+0.13%) |
:arrow_up: |
| ...ython-sdk/src/astro/sql/operators/base_operator.py | 100.00% <100.00%> (ø) |
|
| python-sdk/src/astro/sql/operators/cleanup.py | 97.77% <100.00%> (+0.02%) |
:arrow_up: |
| python-sdk/src/astro/sql/operators/dataframe.py | 93.24% <100.00%> (+0.18%) |
:arrow_up: |
| python-sdk/src/astro/sql/operators/drop.py | 100.00% <100.00%> (ø) |
|
| python-sdk/src/astro/sql/operators/export_file.py | 93.93% <100.00%> (ø) |
|
| python-sdk/src/astro/sql/operators/load_file.py | 97.14% <100.00%> (ø) |
|
| python-sdk/src/astro/sql/operators/merge.py | 100.00% <100.00%> (ø) |
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.