Unable to configure variables with dashes in the name when using GSM secret backend

[odaneau-astro]

Apache Airflow version

2.3.4

What happened

I am attempting to configure variables with a Google Secrets Manager secrets backend and it appears that having dashes in the name prevents Airflow from loading the variables.

Here is what I did:

1. Create some test variables with

   gcloud secrets create airflow-variables-test --replication-policy="automatic"
   gcloud secrets create airflow-variables-test-test --replication-policy="automatic"
   gcloud secrets create airflow-variables-test_test --replication-policy="automatic"
   

2. Logged in the GCP console and set all the secrets values to "12345"
3. Created a simple DAG to test:

   from airflow import DAG
   from airflow.models import Variable
   from airflow.operators.python import PythonOperator
   from datetime import datetime
   
   def print_var():
       variables = ["test", "test-test", "test_test"]
       for var in variables:
           try:
               val = Variable.get(var)
               print(f'Variable {var} is: {val}')
           except Exception:
               print(f"Could not get variable {var}")
   
   with DAG('gsm_secrets_backend', start_date=datetime(2022, 1, 1), schedule_interval=None) as dag:
   
     test_task = PythonOperator(
         task_id='print_var',
         python_callable=print_var
   )
   

4. Ran the DAG locally through astro cli.
5. Looked at the logs:

   [2022-09-16, 14:31:19 UTC] {logging_mixin.py:115} INFO - Variable test is: 12345
   [2022-09-16, 14:31:20 UTC] {secret_manager_client.py:78} ERROR - Google Cloud API Call Error (NotFound): Secret ID airflow-variables-test-test not found.
   [2022-09-16, 14:31:20 UTC] {logging_mixin.py:115} INFO - Could not get variable test-test
   [2022-09-16, 14:31:20 UTC] {logging_mixin.py:115} INFO - Variable test_test is: 12345
   [2022-09-16, 14:31:20 UTC] {python.py:173} INFO - Done. Returned value was: None
   

What you think should happen instead

Variables with dashes in their name should successfully be added

How to reproduce

astro dev init a fresh project

Dockerfile: FROM quay.io/astronomer/astro-runtime:5.0.7

.env:

AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "gcp_keyfile_dict": <service-acct-json-keyfile>}'

astro dev start

Create some test variables with

gcloud secrets create airflow-variables-test-test --replication-policy="automatic"

Log in the GCP console and set the secret value

Create this simple DAG

from airflow import DAG
from airflow.models import Variable
from airflow.operators.python import PythonOperator
from datetime import datetime

def print_var():
    variables = ["test-test"]
    for var in variables:
        try:
            val = Variable.get(var)
            print(f'Variable {var} is: {val}')
        except Exception:
            print(f"Could not get variable {var}")

with DAG('gsm_secrets_backend', start_date=datetime(2022, 1, 1), schedule_interval=None) as dag:

  test_task = PythonOperator(
      task_id='print_var',
      python_callable=print_var
)

Trigger the DAG and look at the logs.

Operating System

Ubuntu 20.04.5 LTS

Versions of Apache Airflow Providers

apache-airflow-providers-google==8.3.0

Deployment

Astronomer

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Feel free to attemp to fix it

[odaneau-astro]

Could not reproduce in Airflow 2.4.