vars set with `deploy` not used by `run`

[kyle-kent]

Describe the issue

This may not be a bug but I couldn't find supporting documentation on this behavior.

When variables that do not have a default value are set with --var at deploy time, they do not carry over to run. Instead the below error is returned

Error: no value assigned to required variable <variable> ...

The variable does indeed get set in the job so I'd expect run to know the value was already set.

Configuration

Create a job that uses the below variable

databricks.yml:

bundle:
  name: dash_2

include:
  - ./resources/*.yaml

variables:
  my_variable:
    description: no default value

targets:
  prod:
    mode: production
    workspace: 
      host: <your host>
      root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${bundle.target}

Steps to reproduce the behavior

1. run databricks bundle deploy -t prod --var="my_variable=new_value"
2. Run databricks bundle run -t prod <job_name>
3. See error

Expected Behavior

If a non-default var is set at deploy time then run will know that and pass without the var set there as well. It should act as you triggered the job from the UI.

Actual Behavior

run does not know the vars set at deploy time.

OS and CLI version

macos and ubuntu cli v0.239.1

Is this a regression?

No

Debug Logs

11:10:33 DEBUG Apply pid=80996 mutator=load mutator=seq
11:10:33 DEBUG Apply pid=80996 mutator=load mutator=seq mutator=EntryPoint
11:10:33 DEBUG Apply pid=80996 mutator=load mutator=seq mutator=scripts.preinit
11:10:33 DEBUG No script defined for preinit, skipping pid=80996 mutator=load mutator=seq mutator=scripts.preinit
11:10:33 DEBUG Apply pid=80996 mutator=load mutator=seq mutator=ProcessRootIncludes
11:10:33 DEBUG Apply pid=80996 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq
11:10:33 DEBUG Apply pid=80996 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq mutator=ProcessInclude(resources/dash_1_scoping.yaml)
11:10:33 DEBUG Apply pid=80996 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq mutator=ProcessInclude(resources/pt__monthly_update.yaml)
11:10:33 DEBUG Apply pid=80996 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq mutator=ProcessInclude(resources/pt__new_cohort.yaml)
11:10:33 DEBUG Apply pid=80996 mutator=load mutator=seq mutator=VerifyCliVersion
11:10:33 DEBUG Apply pid=80996 mutator=load mutator=seq mutator=EnvironmentsToTargets
11:10:33 DEBUG Apply pid=80996 mutator=load mutator=seq mutator=ComputeIdToClusterId
11:10:33 DEBUG Apply pid=80996 mutator=load mutator=seq mutator=InitializeVariables
11:10:33 DEBUG Apply pid=80996 mutator=load mutator=seq mutator=DefineDefaultTarget(default)
11:10:33 DEBUG Apply pid=80996 mutator=load mutator=seq mutator=PythonMutator(load)
11:10:33 DEBUG Apply pid=80996 mutator=load mutator=seq mutator=validate:unique_resource_keys
11:10:33 DEBUG Apply pid=80996 mutator=load mutator=seq mutator=SelectTarget(dev)
11:10:33 DEBUG Loading profile dev because of host match pid=80996
11:10:33 DEBUG Apply pid=80996 mutator=<func>
11:10:33 DEBUG Apply pid=80996 mutator=initialize
11:10:33  INFO Phase: initialize pid=80996 mutator=initialize
11:10:33 DEBUG Apply pid=80996 mutator=initialize mutator=seq
11:10:33 DEBUG Apply pid=80996 mutator=initialize mutator=seq mutator=validate:AllResourcesHaveValues
11:10:33 DEBUG Apply pid=80996 mutator=initialize mutator=seq mutator=RewriteSyncPaths
11:10:33 DEBUG Apply pid=80996 mutator=initialize mutator=seq mutator=SyncDefaultPath
11:10:33 DEBUG Apply pid=80996 mutator=initialize mutator=seq mutator=SyncInferRoot
11:10:33 DEBUG Apply pid=80996 mutator=initialize mutator=seq mutator=PopulateCurrentUser
11:10:34 DEBUG GET /api/2.0/preview/scim/v2/Me
< HTTP/2.0 200 OK
< {
<   "active": true,
<   "emails": [
<     {
<       "primary": true,
<       "type": "work",
<     }
<   ],
<   "groups": [
<     {
<     },
<     {
<     },
<     {
<     }
<   ],
<   "id": "8793528040177278",
<   "name": {
<   },
<   "roles": [
<     {
<   ],
<   "schemas": [
<     "urn:ietf:params:scim:schemas:core:2.0:User",
<     "... (1 additional elements)"
<   ],
< } pid=80996 mutator=initialize mutator=seq mutator=PopulateCurrentUser sdk=true
11:10:34 DEBUG Apply pid=80996 mutator=initialize mutator=seq mutator=LoadGitDetails
11:10:34 DEBUG Apply pid=80996 mutator=initialize mutator=seq mutator=ApplySourceLinkedDeploymentPreset
11:10:34 DEBUG Apply pid=80996 mutator=initialize mutator=seq mutator=DefineDefaultWorkspaceRoot
11:10:34 DEBUG Apply pid=80996 mutator=initialize mutator=seq mutator=ExpandWorkspaceRoot
11:10:34 DEBUG Apply pid=80996 mutator=initialize mutator=seq mutator=DefaultWorkspacePaths
11:10:34 DEBUG Apply pid=80996 mutator=initialize mutator=seq mutator=PrependWorkspacePrefix
11:10:34 DEBUG Apply pid=80996 mutator=initialize mutator=seq mutator=RewriteWorkspacePrefix
11:10:34 DEBUG Apply pid=80996 mutator=initialize mutator=seq mutator=SetVariables
11:10:34 ERROR Error: no value assigned to required variable DBT_PROJECT_SCHEMA. Assignment can be done using "--var", by setting the BUNDLE_VAR_DBT_PROJECT_SCHEMA environment variable, or in .databricks/bundle/<target>/variable-overrides.json file pid=80996 mutator=initialize mutator=seq mutator=SetVariables
11:10:34 ERROR Error: no value assigned to required variable DBT_PROJECT_SCHEMA. Assignment can be done using "--var", by setting the BUNDLE_VAR_DBT_PROJECT_SCHEMA environment variable, or in .databricks/bundle/<target>/variable-overrides.json file pid=80996 mutator=initialize mutator=seq
11:10:34 ERROR Error: no value assigned to required variable DBT_PROJECT_SCHEMA. Assignment can be done using "--var", by setting the BUNDLE_VAR_DBT_PROJECT_SCHEMA environment variable, or in .databricks/bundle/<target>/variable-overrides.json file pid=80996 mutator=initialize
Error: no value assigned to required variable DBT_PROJECT_SCHEMA. Assignment can be done using "--var", by setting the BUNDLE_VAR_DBT_PROJECT_SCHEMA environment variable, or in .databricks/bundle/<target>/variable-overrides.json file
11:10:34 ERROR failed execution pid=80996 exit_code=1 error="no value assigned to required variable DBT_PROJECT_SCHEMA. Assignment can be done using \"--var\", by setting the BUNDLE_VAR_DBT_PROJECT_SCHEMA environment variable, or in .databricks/bundle/<target>/variable-overrides.json file"

[github-actions[bot]]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[pietern]

Thanks for posting the issue.

Can you clarify how you are using these variables? Do they have default values, and how do you vary them for different deployments? Is it possible to export them as environment variables to persist their value across different commands?

[kyle-kent]

@pietern they're like env vars. There's no default value. They're deployed typically by gh actions which hardcodes them per env. Then run can be triggered by other gh actions if needed - it's more convenient than an api call.

[solita-robertofierimonte]

👍 I'd also be interested in seeing this implemented