Example of a job with tasks that run other jobs

[pietern]

trafficstars

Note: this requires https://github.com/databricks/cli/pull/1219.

[cristian-rincon]

I encountered multiple errors while testing this approach. It appears that variable substitutions do not work with this level of nesting. Do you have any suggestions or workarounds to fix this issue?

Traceback

❯ databricks bundle deploy
Building default...
Uploading sample-1.80.2-py3-none-any.whl...
Uploading bundle files to /Users/myuser/.bundle/sample_core/local/files...
Deploying resources...
Updating deployment state...
Error: terraform apply: exit status 1

Error: cannot create job: Missing required field: settings.tasks.task_key.

  with databricks_job.sample_mlops_main_stage,
  on bundle.tf.json line 56, in resource.databricks_job.sample_mlops_main_stage:
  56:       },


Error: cannot update job: Missing required field: new_settings.webhook_notifications.on_failure.id

  with databricks_job.sample_mlops_stage_1__models_training_and_serving,
  on bundle.tf.json line 1362, in resource.databricks_job.sample_mlops_stage_1__models_training_and_serving:
1362:       },

Config files

databricks.yml

...
variables:
  webhook_notifications_id:
    description: The ID of the webhook notification to use for the job.
    default: 'xzy-xyz-uxz'
  training_and_serving_job1_job_id:
    default: ''

sample_job.yml

...
resources:
  jobs:
    sample_mlops_stage_1__models_training_and_serving:
      name: 'sample_mlops_stage_1__models_training_and_serving'
      tasks:
        - task_key: job1_id
          libraries:
            - whl: ../dist/*.whl
          run_if: ALL_SUCCESS
          email_notifications: {}
          run_job_task:
            job_id: ${var.job1_id}
...
      webhook_notifications:
        on_failure:
          - id: ${var.webhook_notifications_id}

[pietern]

@cristian-rincon Which version of the CLI are you using? Support for this was released in v0.214.0.

[cristian-rincon]

@cristian-rincon Which version of the CLI are you using? Support for this was released in v0.214.0.

I'm using Databricks CLI v0.213.0, i will test with v0.214.0

[cristian-rincon]

@cristian-rincon Which version of the CLI are you using? Support for this was released in v0.214.0.

I'm using Databricks CLI v0.213.0, i will test with v0.214.0

@pietern I have upgraded the CLI and now i got this error:

Traceback

❯ databricks bundle deploy
Building default...
Uploading sample-1.80.2-py3-none-any.whl...
Uploading bundle files to /Users/myuser/.bundle/sample_core/local/files...
Deploying resources...
Updating deployment state...
Error: terraform apply: exit status 1

Error: Missing required argument

  on bundle.tf.json line 51, in resource.databricks_job.sample_mlops_main_stage.task[0].run_job_task:
  51:             "run_job_task": {},

The argument "job_id" is required, but no definition was found.

main.yml

resources:
  jobs:
    sample_mlops_main_stage:
      name: 'sample_mlops_main_stage'
      tasks:
        - task_key: abt
          run_job_task:
            # job_id: 234902340127014
            job_id: ${resources.jobs.sample_mlops_stage_0__job1.id}
          email_notifications: {}
          libraries:
            - whl: ../dist/*.whl
          run_if: ALL_SUCCESS

[pietern]

Could you post this as an issue on the CLI repository, and include the relevant bits of your bundle validate output.

The YAML looks valid if it is used verbatim. Note that a libraries section is not relevant for a run_job_task.

[cristian-rincon]

Could you post this as an issue on the CLI repository, and include the relevant bits of your bundle validate output.

The YAML looks valid if it is used verbatim. Note that a libraries section is not relevant for a run_job_task.

@pietern I changed the job name by removing the double underscore, and it fixed the issue. for ex:

• Non working solution : sample_mlops_stage_0__job1
• Working solution: sample_mlops_stage_0_job1