dbx feature: provide an option to deploy init_scripts as workspace files via dbx

feature: provide an option to deploy init_scripts as workspace files via dbx

Open domste opened this issue 2 years ago • 1 comments

trafficstars

Expected Behavior

Provide an option to upload (and use) cluster init-scripts to the workspace via deployment.yaml. Storing them in dbfs is deprecated (see https://docs.databricks.com/api/azure/workspace/clusters/create).

Current Behavior

It is not possible to automatically upload files referenced in deployment.conf using either file:// or file:fuse;// as a workspace file. Which on the other hand is suggested as the new best practice.

Also there is no hint that using cluster-scoped init-scripts is the way to go in the examples here

Steps to Reproduce (for bugs)

Context

We are heavily using job clusters with deployments via deployment.yaml and as init-scripts located in dbfs are not supported any more from 1st of September there should be a solution to tackle this problem using dbx .

Workspace announcement

DB clusters create API

Deprecation note on cluster-scoped init scripts in dbfs

https://learn.microsoft.com/en-us/azure/databricks/files/workspace-init-scripts