astro-sdk
astro-sdk copied to clipboard
Allow `aql.render` to use the DAG's `template_searchpath`
Context
When instantiating Airflow DAGs, users can declare template_searchpath, which is a list of folders (non-relative) that define where jinja will look for templates. By default, it contains the DAGs folder.
Many SQL-like operators, such as the SnowflakeOperator and BigQueryOperator, support retrieving SQL files from this path, as described in the doc.
At the moment, Astro 0.5.1 expects users to declare the path to the SQL files as an argument of the aql.render method.
Acceptance criteria
aql.rendertries to fetch SQL files from each of the DAG'stemplate_searchpathvariables.- If a user supplies a "path" variable,
aql.renderwill check each template searchpath for that subpath (so if template_searchpath =['foo/bar', 'fizz/buzz'] and path='baz', we will search ['foo/bar/baz', 'fizz/buzz/baz'] - If there are no matches for directories in any of those template_searchpaths, we will return the original path and assume it is an absolute path.
- Ask @mag3141592 (bug reporter) to review the PR
@ashb @tatiana I think there are a few "gotchas" in this one.
What do we do if there are multiple values in the template_searchpath? Wouldn't this only work once? Like we wouldn't be able to run render mutliple times unless we're point to subdirectories? If we're pointing to subdirectories how do we know what order to do them in?
Is this still relevant @tatiana ?