Support Airflow Variable?

[everglory99]

Hi, Thanks for open sourcing boundary-layer, this is really useful! While I am prototyping some plugins for some other open sourced operators and internal operators, I can't find any support for Variables (https://airflow.apache.org/concepts.html#variables). Am I missing something? Is there a plan to support Variable? The use case is that we want to maintain one yaml file across different environments. Thanks for the help!

[mchalek]

hi @everglory99 ! I'm glad to hear that you have found boundary-layer useful.

We do not currently have first-class support for variables, although you can in principle access them using the "verbatim string" approach. It would require specifying that the DAG import the Variable class, and it would also only work right now for operator properties that accept strings (as opposed to integers etc). For example, your workflow might look something like:

name: my-workflow

imports:
  objects:
  - module: airflow.model
    object: Variable

operators:
- name: my-operator
  type: my-operator-type
  properties:
    my_arg: <<Variable.get("variable_name")>>

Or alternatively you could access them via the airflow jinja templates, as shown in the link you included, again with the requirement that the operator property be of type string.

I hope that one of these approaches will be useful to you now, but please let us know if it is not. We are gradually assembling a feature wish-list for the next significant version bump, and this would certainly be a viable candidate. We don't use airflow Variables at Etsy so we're not currently prioritizing it, but we're excited to have external interest in this project, so if you need broader support for variables, we'll factor that into our considerations :)

[everglory99]

Thanks so much for your help, I totally miss the "verbatim string" part in your blog. After fixing some typos in your code example, the following works great for me!

name: my-workflow

imports:
  objects:
    - module: airflow.models
      objects: 
        - Variable

operators:
- name: my-operator
  type: my-operator-type
  properties:
    my_arg: <<Variable.get("variable_name")>>

boundary-layer fits perfectly in some of our use cases. I am excited to see this project grow!

[mchalek]

Ooh, great, I'm glad to hear it @everglory99 ! And sorry for the typos, it was late when I wrote it 😉. Thanks for including the corrected example.

I will leave this issue open as a reminder to include airflow Variable as something we should consider providing more official support for.

[istphc]

I still hope official Variable support available due to the limitation of nested Verbatim usage.

For example,

operators:
- name: my-operator
  type: kubernetes
  properties:
    secrets: 
    - deploy_target: <<Variable.get("deploy_target")>>
       deploy_type: <<Variable.get("deploy_type")>>
       secret: <<Variable.get("secret")>>

is not possible because secrets itself is using Verbatim.