Execution failure - insecure property disregarded

[bb831]

trafficstars

Expected Behavior

Use insecure property from profile provided by .databrickscfg file when executing mlflow

Current Behavior

Uses system environmental variables set in a class instead of profile.

Steps to Reproduce (for bugs)

dbx execute --cluster-id=cluster_id file_name --no-package

Context

When running the dbx execute command the request is disregarding the insecure property set in the .databrickscfg file.

In the file mlflow_based.py the class MlflowStorageConfigurationManager sets only the class variables DATABRICKS_HOST_ENV and DATABRICKS_TOKEN_ENV. Since there is no insecure property check from the .databrickscfg file and a corresponding class ENV variable is not set this causes the downstream failure. When the function getconfig() from class DefaultConfigProvider(DatabricksConfigProvider) is called later located in provider.py. The function loops through locations where credentials can be found in the following order:

1. SparkTaskContextConfigProvider
2. EnvironmentVariableConfigProvider
3. ProfileConfigProvider

Since a profile is being provided by a user through a .databrickscfg file and in that file the insecure property is being set the expectation would be that the third option is used, but the execution fails because the loop stops at the second option. This is due to host/token environmental variables being set in MlflowStorageConfigurationManager mentioned earlier and not including the insecure property if needed.

Suggested resolution:

• Changing the order in which the getconfig() function mentioned so that profile is checked prior to environment in class DefaultConfigProvider(DatabricksConfigProvider)

OR

• Adding a check set cls insecure variable in MlflowStorageConfigurationManager

Trying both options independently resolved execution failure.

Your Environment

Windows Pycharm

• dbx version used: 0.8.10
• Databricks Runtime version: 11.3