Cannot deploy without a default value in Variable.get()

[kushalmalani]

I had a DAG with Variable.get() without a default value, everything worked as expected locally. However, I saw an error while deploying to Astro: ValueError: invalid literal for int() with base 10: 'NON_DEFAULT_MOCKED_VARIABLE_VALUE'. After adding default value (Variable.get('my_var', default_value=50)) deployment was successful. Why a user is forced to add a default value?

Original Ticket - https://github.com/astronomer/cloud-cli/issues/289

[alex-astronomer]

Hi @magdagultekin, I'm not able to recreate this error. When I call Variable.get("something") and my something keyed variable exists, there are no errors found. Not sure if the existence of this key matters at this point, I would assume not.

Will you post the DAG here that is showing errors?

Thanks!

[magdagultekin]

Sure @alex-astronomer, please see below. I saw a similar problem in on of the channels (but can't find it now :[) where the deployment wasn't successful although there were no issues found - maybe you also saw it?

from datetime import datetime, timedelta

from airflow import DAG
from airflow.decorators import task
from airflow.models import Variable


default_args = {
        'owner': 'cs',
        'depends_on_past': False,
        'email_on_failure': False,
        'email_on_retry': False,
        'retries': 2,
        'retry_delay': timedelta(minutes=5),
    }


with DAG(dag_id='dynamic_task_mapping_known_values',
         start_date=datetime(2022, 5, 1),
         schedule_interval='@once',
         default_args=default_args,
         tags=['dynamic tak mapping', 'aws parameter store', 'secrets backend'],
         description='''
            This DAG demonstrates dynamic task mapping with a constant parameter based on the known values, 
            and retrieving variables from AWS Parameter Store.
         ''',
         ) as dag:

    @task
    def add(x, y):
        return x + y

    # partial(): allows you to pass parameters that remain constant for all tasks
    # expand(): allows you to pass parameters to map over
    added_values = add.partial(x=int(Variable.get("my_test_variable"))).expand(y=[0, 1, 2])
    # Variable comes from the AWS Parameter Store

After I added default value (Variable.get("my_test_variable", 50)), deployment was completed.

[sunkickr]

@dylanbstorey do you know of any reason why a default variable would be needed for this DAG to parse?

[alex-astronomer]

I can look into this tuesday