astro-sdk icon indicating copy to clipboard operation
astro-sdk copied to clipboard

Allow `aql.render` to use the DAG's `template_searchpath`

Open tatiana opened this issue 3 years ago • 1 comments
trafficstars

Context When instantiating Airflow DAGs, users can declare template_searchpath, which is a list of folders (non-relative) that define where jinja will look for templates. By default, it contains the DAGs folder.

Many SQL-like operators, such as the SnowflakeOperator and BigQueryOperator, support retrieving SQL files from this path, as described in the doc.

At the moment, Astro 0.5.1 expects users to declare the path to the SQL files as an argument of the aql.render method.

Acceptance criteria

  • aql.render tries to fetch SQL files from each of the DAG's template_searchpath variables.
  • If a user supplies a "path" variable, aql.render will check each template searchpath for that subpath (so if template_searchpath =['foo/bar', 'fizz/buzz'] and path='baz', we will search ['foo/bar/baz', 'fizz/buzz/baz']
  • If there are no matches for directories in any of those template_searchpaths, we will return the original path and assume it is an absolute path.
  • Ask @mag3141592 (bug reporter) to review the PR

tatiana avatar Feb 21 '22 11:02 tatiana

@ashb @tatiana I think there are a few "gotchas" in this one.

What do we do if there are multiple values in the template_searchpath? Wouldn't this only work once? Like we wouldn't be able to run render mutliple times unless we're point to subdirectories? If we're pointing to subdirectories how do we know what order to do them in?

dimberman avatar Mar 05 '22 01:03 dimberman

Is this still relevant @tatiana ?

sunank200 avatar Jan 19 '23 09:01 sunank200