astro-sdk icon indicating copy to clipboard operation
astro-sdk copied to clipboard

Easier access to the Airflow context

Open rossturk opened this issue 3 years ago • 5 comments

Please describe the feature you'd like to see

For getting Airflow context, get_current_context() is supported, and it seems to work fine. However, I had to search through the Airflow docs to find it and the resulting code is a bit wordy:

@aql.dataframe
def do_stuff():
    context = get_current_context()
    data_interval_start = context["data_interval_start"]
    data_interval_end = context["data_interval_end"]
	  ...

I think this is something that pipeline builders will need to do on every pipeline that can backfill, which should be a lot of 'em.

Describe the solution you'd like

Could context be available via kwargs like they are in TaskFlow?

@aql.dataframe
def do_stuff(data_interval_start=None,data_interval_end=None):
    ...

Are there any alternatives to this feature?

We could document get_current_context() as the correct way to do this in the Astro SDK docs.

Acceptance Criteria

  • [ ] All checks and tests in the CI should pass
  • [ ] Unit tests (90% code coverage or more, once available)
  • [ ] Integration tests (if the feature relates to a new database or external service)
  • [ ] Example DAG
  • [ ] Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
  • [ ] Exception handling in case of errors
  • [ ] Logging (are we exposing useful information to the user? e.g. source and destination)
  • [ ] Improve the documentation (README, Sphinx, and any other relevant)
  • [ ] How to use Guide for the feature (example)

rossturk avatar Sep 09 '22 05:09 rossturk

@feluelle ^^

utkarsharma2 avatar Sep 12 '22 10:09 utkarsharma2

@rossturk Thanks for creating this ticket, we'll take a look at it. :)

utkarsharma2 avatar Sep 12 '22 10:09 utkarsharma2

@kaxil any thoughts on how critical this is ? Considering the backlog we have at the moment, I dont think we can work on this atleast till next sprint

phanikumv avatar Jan 16 '23 12:01 phanikumv

@phanikumv - low

kaxil avatar Jan 16 '23 18:01 kaxil

As agreed with @jlaneve, we're deferring this to the milestone 1.7.

tatiana avatar May 03 '23 13:05 tatiana