astro-sdk
astro-sdk copied to clipboard
Easier access to the Airflow context
Please describe the feature you'd like to see
For getting Airflow context, get_current_context() is supported, and it seems to work fine. However, I had to search through the Airflow docs to find it and the resulting code is a bit wordy:
@aql.dataframe
def do_stuff():
context = get_current_context()
data_interval_start = context["data_interval_start"]
data_interval_end = context["data_interval_end"]
...
I think this is something that pipeline builders will need to do on every pipeline that can backfill, which should be a lot of 'em.
Describe the solution you'd like
Could context be available via kwargs like they are in TaskFlow?
@aql.dataframe
def do_stuff(data_interval_start=None,data_interval_end=None):
...
Are there any alternatives to this feature?
We could document get_current_context() as the correct way to do this in the Astro SDK docs.
Acceptance Criteria
- [ ] All checks and tests in the CI should pass
- [ ] Unit tests (90% code coverage or more, once available)
- [ ] Integration tests (if the feature relates to a new database or external service)
- [ ] Example DAG
- [ ] Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
- [ ] Exception handling in case of errors
- [ ] Logging (are we exposing useful information to the user? e.g. source and destination)
- [ ] Improve the documentation (README, Sphinx, and any other relevant)
- [ ] How to use Guide for the feature (example)
@feluelle ^^
@rossturk Thanks for creating this ticket, we'll take a look at it. :)
@kaxil any thoughts on how critical this is ? Considering the backlog we have at the moment, I dont think we can work on this atleast till next sprint
@phanikumv - low
As agreed with @jlaneve, we're deferring this to the milestone 1.7.