Unable to use standard Google conn_id for aql.load_file into BigQuery

[TheCodyRich]

trafficstars

Describe the bug When using aql.load_file to load data from GCS to BQ, task fails due to authentication error. The GCSToBigQueryOperator is able to run successfully using the Google Cloud Platform connection, however the astro sdk does not. I was able to get the astro sdk task to run successfully by adding RUN curl -sSL https://sdk.cloud.google.com | bash to the docker file then manually authenticating in the docker container and running gcloud auth application-default login (i.e. not using the conn_id).

Version

• Astro SDK 0.11.0
• Astro CLI 1.02
• Astro run-time 5.0.6

To Reproduce Steps to reproduce the behavior:

1. Create a google cloud connection utilizing the connection type, "Google Cloud Platform" (Figure 1). Message me for connection details
2. Run attached DAG as is with connection details provided by me, or if using your own connection details update paths, schemas, and variables to reflect your own data and connections
3. See the astro sdk task fails due to an auth error while the GCSToBigQueryOperator tasks run successfully (Figure 2)

Expected behavior astro sdk should be able to authenticate to Google BigQuery by providing the Google Cloud Platform connection id using the conn_id google_cloud_default

Screenshots

Figure 1: Google Cloud connection

Figure 2: astro sdk task fails (due to auth error) with GCS BQ operators succeed.

Additional context Full logs from the astro sdk task (gcs_to_bq_sdk)

test DAG