astro-sdk icon indicating copy to clipboard operation
astro-sdk copied to clipboard

Implement `get_value_list` (Dynamic Task Templates)

Open kaxil opened this issue 1 year ago • 0 comments

This builds on top of Dynamic Tasks Mapping released in Airflow 2.3 and is similar to https://github.com/astronomer/astro-sdk/issues/507

get_value_list(‘select product from products’, db_conn) 

Iterating over a set of files matching a pattern in a file system:

  • Query a table in a database to get a list of values.
  • And then, to iterate over a set of tasks for each value in this list.
  • The number of values in this list is unknown at the time of DAG creation, but it is not uncommon for this list to be hundreds of values.

The get_value_list operation would abstract away the underlying Operator setup and boilerplate code, similar to the existing load_file operation in the Astro Python SDK.

This would enable the DAG author to get a list of products from the database, accessed using the specified connection. In the spirit of the existing DAG authoring framework, the underlying API takes care of all the boilerplate for getting the right database Operator, setting it up, executing the SQL, and getting the data back into a list suitable for the ensuing “map” operation.

kaxil avatar Aug 15 '22 18:08 kaxil