bundle-examples icon indicating copy to clipboard operation
bundle-examples copied to clipboard

Inaccurate reference to Databricks.yml?

Open johalnes opened this issue 1 year ago • 3 comments

Hi,

I'm working on deploying a DBT project trough Databricks Asset bundles, but somewhat confused on how to get environment variables into the task. In this example, how is {{ env_var("DBT_ACCESS_TOKEN") }} defined as an environment variable?

Is the databricks.yml configuration working as it is now? I get it to work without passing environment variables, but setting job parameters doesn't seem to work.

Would be super grateful for some help 🙂

johalnes avatar Apr 25 '24 09:04 johalnes

Hey @johalnes you will need to create an environment variable for your GitHub repo by going to settings > secrets and variables > actions

Create a variable called DBT_ACCESS_TOKEN and set it to your Databricks PAT

{{ env_var('DBT_ACCESS_TOKEN') }} specifically parses a GitHub Actions env var

more info on PAT auth

PatrickLeahey avatar May 01 '24 21:05 PatrickLeahey

Thanks for replying @PatrickLeahey ! But not really what I was wondering about.

I know how secrets and environment variables within Github works. But what is the link between environment variables and how can one use these within Workflows? I've created a environment variable {{ env_var("DBT_SCHEMA") }}. And even when defined as workflow parameters, and defined in my environment where I run databricks bundle deploy , Databricks Workflows don't seem to get this variable.

How would you defined this without using --vars "{ dev_schema: ${workspace.current_user.short_name} }"' like in this example repo?

johalnes avatar May 06 '24 09:05 johalnes

The inaccurate part of the headline was regarding you writing :

# The workspace host / token are provided by Databricks
# see databricks.yml for the host used for 'dev' 

in the code, without any references to these variables within the actual databricks.yml file 🙂

johalnes avatar May 06 '24 09:05 johalnes