dag-factory
dag-factory copied to clipboard
How to access Airflow variables and parameters passed via REST API
I have a use case where I am accessing Airflow's variable using Variable.get(). How can I achieve this in yaml file.
Also, how can I access parameters (params.app_name, params.namespace etc) which I am passing via REST API as follows:
{ "conf": { "app_name":"app namet", "region":"", "namespace": "", "default_path": "", } }
env_variables = Variable.get("env_variables", deserialize_json=True)
fetch_yaml = BashOperator(
bash_command="export {{var.json.env_variables['kube_config'] && kubectl get sparkapplications {{ params.app_name}} -n {{params.namespace}}",
task_id="fetch_yaml"
)
Airflow variables can be accessed as below
within yaml specify any attribute with the value as shown below. Note the quotes, some of them are two single quotes. ENV: '{{ var.value.get(''ENVIRONMENT'', ''no-mans-land'') }}'
I am also looking at how to access params though, which is typically as
DAG(params={...}), however specifying these with in yaml does not resolve.
YAML
params:
key: value
comes out as below when printing ... print(vars(context.get('dag')))
'params': {}
Reference : https://github.com/apache/airflow/blob/main/airflow/models/dag.py#L229
PR submitted.
https://github.com/ajbosco/dag-factory/pull/124