astro-sdk icon indicating copy to clipboard operation
astro-sdk copied to clipboard

Create `upstream_tasks` parameter for dependencies independent of data transfers

Open dimberman opened this issue 3 years ago • 1 comments

Description

What is the current behavior?

Currently there is "native" way to set dependencies between tasks without passing either a table or a dataframe between said tasks. This is especially apparent when chaining raw_sql functions such as here:

https://github.com/astronomer/astro-sdk/issues/524

In order to have a solution that works we need somehting that feels pythonic and doesn't rely on traditional airflow >> operators

closes: #524

What is the new behavior?

This PR adds in the ability to set dependencies using a parameter upstream_tasks. This parameter can be used at runtime to set deps without passing data. Notably we don't want to actually render the data (imagine a use-case where a task relies on 10 dataframe tasks)

Does this introduce a breaking change?

No

Checklist

  • [ ] Created tests which fail without the change (if possible)
  • [ ] Extended the README / documentation, if necessary
  • [ ] Add documentation around upstream_tasks
  • [ ] solution should not render the data since the data is never used.

dimberman avatar Jul 28 '22 21:07 dimberman

Codecov Report

Merging #585 (a5a6e9f) into main (62303bd) will decrease coverage by 0.13%. The diff coverage is 88.57%.

@@            Coverage Diff             @@
##             main     #585      +/-   ##
==========================================
- Coverage   93.26%   93.12%   -0.14%     
==========================================
  Files          43       45       +2     
  Lines        1855     1877      +22     
  Branches      232      237       +5     
==========================================
+ Hits         1730     1748      +18     
- Misses         97      100       +3     
- Partials       28       29       +1     
Impacted Files Coverage Δ
...sdk/src/astro/sql/operators/upstream_task_mixin.py 69.23% <69.23%> (ø)
python-sdk/src/astro/sql/operators/append.py 100.00% <100.00%> (ø)
...thon-sdk/src/astro/sql/operators/base_decorator.py 93.13% <100.00%> (+0.13%) :arrow_up:
...ython-sdk/src/astro/sql/operators/base_operator.py 100.00% <100.00%> (ø)
python-sdk/src/astro/sql/operators/cleanup.py 97.77% <100.00%> (+0.02%) :arrow_up:
python-sdk/src/astro/sql/operators/dataframe.py 93.24% <100.00%> (+0.18%) :arrow_up:
python-sdk/src/astro/sql/operators/drop.py 100.00% <100.00%> (ø)
python-sdk/src/astro/sql/operators/export_file.py 93.93% <100.00%> (ø)
python-sdk/src/astro/sql/operators/load_file.py 97.14% <100.00%> (ø)
python-sdk/src/astro/sql/operators/merge.py 100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

codecov[bot] avatar Aug 02 '22 16:08 codecov[bot]