Jobs 2.1 API setting ignored by DatabricksClientProvider

[aholten]

Expected Behavior

Despite this closed issue stating the contrary, dbx does not support the Databricks Jobs 2.1 API.

Current Behavior

The config/environment variable jobs-api-version value is not passed the ApiClient constructor in DatabricksClientProvider's _get_v2_client class method even when available as config.jobs_api_version as provided by config = AuthConfigProvider.get_config().

Steps to Reproduce (for bugs)

• Configure a profile in .databrickscfg to include jobs-api-version = 2.1
• Attempt any dbx command that uses a jobs API endpoint using that profile (eg. dbx deploy)
• More specifically, the incorrect API urls can be revealed with breakpoints or by attempting to deploy an incorrectly specified deployment yaml/json

Context

I would like to use the Jobs 2.1 API for multitask job job cluster sharing.

Your Environment

• dbx version used: 0.6.10
• Databricks Runtime version: N/A