[BUG]: Can't deploy to databricks anymore

[fwaris]

Describe the bug A clear and concise description of what the bug is.

dotnet spark jobs do not run on the latest version of databricks clusters in Azure. This worked before (about 3 or so months ago) but is no longer working. With the db_init.sh (see attached) based on the one given in the repository there is no log output of any kind. Without db_init.sh there is an error in the stderr log:

Caused by: java.lang.NullPointerException
	at com.databricks.backend.daemon.driver.WsfsDriverHttpClient.<init>(WSFSDriverHttpClient.scala:26)
	at com.databricks.backend.daemon.driver.WSFSCredentialForwardingHelper$.<init>(WorkspaceLocalFileSystem.scala:284)
	at com.databricks.backend.daemon.driver.WSFSCredentialForwardingHelper$.<clinit>(WorkspaceLocalFileSystem.scala)
	... 28 more

Steps to reproduce the behavior:

1. create a simple dotnet spark job that reads a csv file and shows the results
2. publish the job
3. deploy the job to databricks cluser
4. run the job
5. note the job fails and there is no log output of any kind

Expected behavior The job should not fail db_init.zip

[fwaris]

updated (please ignore previous message - I had specified the script name the name wrong)

The only error I can see is this:

Unexpected failure while waiting for the cluster (0616-173505-bj32g43i) to be ready.Cause Unexpected state for cluster (0616-173505-bj32g43i): INIT_SCRIPT_FAILURE(CLIENT_ERROR): instance_id:ff51a946a7ea4ab5a167e11ebc455def,databricks_error_message:Cluster scoped init script dbfs:/FileStore/faisal/db-init.sh failed: Script exit status is non-zero

Looks like the init script is failing but there is no way to debug this on databricks.

@luisquintanilla