[ISSUE] `databricks_external_location` incorrectly propagates its `url` attribute

[camilo-s]

Configuration

terraform {
  required_providers {
    databricks = {
      configuration_aliases = [
        databricks.account,
        databricks.default
      ]
      source = "databricks/databricks"
    }
  }
}

resource "databricks_metastore_assignment" "this" {
  metastore_id = var.metastore_id
  workspace_id = var.databricks_workspace_id

  provider = databricks.default
}

resource "databricks_storage_credential" "this" {
  name  = var.access_connector_name
  owner = var.client_principal_id
  azure_managed_identity {
    access_connector_id = var.access_connector_id
  }

  depends_on = [
    databricks_metastore_assignment.this
  ]

  provider = databricks.default
}

resource "databricks_grant" "service_principal" {
  metastore  = var.metastore_id
  principal  = var.client_principal_id
  privileges = ["CREATE_EXTERNAL_LOCATION"]

  depends_on = [
    databricks_metastore_assignment.this
  ]

  provider = databricks.default
}

resource "databricks_external_location" "this" {
  for_each = var.catalogs

  name            = "${each.value.storage_account_name}_${replace(each.value.container_name, "-", "_")}"
  url             = "abfss://${each.value.container_name}@${each.value.storage_account_name}.dfs.core.windows.net"
  credential_name = databricks_storage_credential.this.id
  force_destroy   = true

  depends_on = [
    databricks_grant.service_principal
  ]

  provider = databricks.default
}

resource "databricks_mws_ncc_binding" "this" {
  count                          = var.network_connectivity_config_id != null ? 1 : 0
  network_connectivity_config_id = var.network_connectivity_config_id
  workspace_id                   = var.databricks_workspace_id

  provider = databricks.account
}

resource "databricks_sql_endpoint" "this" {
  name                      = "Test Warehouse"
  cluster_size              = "2X-Small"
  auto_stop_mins            = 10
  enable_serverless_compute = true
  enable_photon             = false
  warehouse_type            = "PRO"

  provider = databricks.default
}

resource "databricks_catalog" "this" {
  for_each = var.catalogs

  name           = each.key
  storage_root   = databricks_external_location.this[each.key].url
  isolation_mode = "ISOLATED"
  force_destroy  = true

  provider = databricks.default
}

Expected Behavior

The catalog get's deployed using the external location's url as storage root.

Actual Behavior

When trying to dynamically set the storage_root argument for a databricks_catalog from via the url attribute of a databricks_external_location, Terraform fails during a first apply, with the following error:

2024-05-23T14:58:37.6770006Z module.test_workspace.time_static.current: Creating...
2024-05-23T14:58:37.6832413Z module.test_workspace.time_static.current: Creation complete after 0s [id=2024-05-23T14:58:37Z]
2024-05-23T14:58:40.6811931Z module.test_workspace.azurerm_resource_group.this: Creating...
2024-05-23T14:58:50.2165093Z module.test_workspace.azurerm_resource_group.this: Creation complete after 9s [id=/subscriptions/***/resourceGroups/rg-dev-testncc-camilo-001]
2024-05-23T14:58:50.2237199Z module.test_workspace.azurerm_databricks_access_connector.this: Creating...
2024-05-23T14:58:50.2237855Z module.test_workspace.azurerm_databricks_workspace.this: Creating...
2024-05-23T14:58:50.2441245Z module.test_workspace.azurerm_storage_account.this: Creating...
2024-05-23T14:59:00.2260392Z module.test_workspace.azurerm_databricks_workspace.this: Still creating... [10s elapsed]
2024-05-23T14:59:00.2261523Z module.test_workspace.azurerm_databricks_access_connector.this: Still creating... [10s elapsed]
2024-05-23T14:59:00.2442557Z module.test_workspace.azurerm_storage_account.this: Still creating... [10s elapsed]
2024-05-23T14:59:03.2189295Z module.test_workspace.azurerm_databricks_access_connector.this: Creation complete after 13s [id=/subscriptions/***/resourceGroups/rg-dev-testncc-camilo-001/providers/Microsoft.Databricks/accessConnectors/dbac-testncc-camilo-001]
2024-05-23T14:59:10.2266376Z module.test_workspace.azurerm_databricks_workspace.this: Still creating... [20s elapsed]
2024-05-23T14:59:10.2448912Z module.test_workspace.azurerm_storage_account.this: Still creating... [20s elapsed]
2024-05-23T14:59:14.3825163Z module.test_workspace.azurerm_storage_account.this: Creation complete after 24s [id=/subscriptions/***/resourceGroups/rg-dev-testncc-camilo-001/providers/Microsoft.Storage/storageAccounts/sttestncccamilo]
2024-05-23T14:59:14.3928739Z module.test_workspace.azurerm_storage_account_network_rules.databricks_serverless_compute[0]: Creating...
2024-05-23T14:59:14.3929648Z module.test_workspace.azurerm_role_assignment.this: Creating...
2024-05-23T14:59:14.3940085Z module.test_workspace.azurerm_storage_container.this: Creating...
2024-05-23T14:59:14.8556990Z module.test_workspace.azurerm_storage_container.this: Creation complete after 1s [id=https://sttestncccamilo.blob.core.windows.net/00-catalog]
2024-05-23T14:59:17.4607074Z module.test_workspace.azurerm_storage_account_network_rules.databricks_serverless_compute[0]: Creation complete after 3s [id=/subscriptions/***/resourceGroups/rg-dev-testncc-camilo-001/providers/Microsoft.Storage/storageAccounts/sttestncccamilo]
2024-05-23T14:59:20.2271800Z module.test_workspace.azurerm_databricks_workspace.this: Still creating... [30s elapsed]
2024-05-23T14:59:24.3933298Z module.test_workspace.azurerm_role_assignment.this: Still creating... [10s elapsed]
2024-05-23T14:59:30.2279225Z module.test_workspace.azurerm_databricks_workspace.this: Still creating... [40s elapsed]
2024-05-23T14:59:34.3940696Z module.test_workspace.azurerm_role_assignment.this: Still creating... [20s elapsed]
2024-05-23T14:59:40.2283423Z module.test_workspace.azurerm_databricks_workspace.this: Still creating... [50s elapsed]
2024-05-23T14:59:40.9521155Z module.test_workspace.azurerm_role_assignment.this: Creation complete after 27s [id=/subscriptions/***/resourceGroups/rg-dev-testncc-camilo-001/providers/Microsoft.Storage/storageAccounts/sttestncccamilo/providers/Microsoft.Authorization/roleAssignments/f9b05034-8d21-8708-eb1f-e9739bb9a552]
2024-05-23T14:59:50.2306616Z module.test_workspace.azurerm_databricks_workspace.this: Still creating... [1m0s elapsed]
2024-05-23T15:00:00.2308795Z module.test_workspace.azurerm_databricks_workspace.this: Still creating... [1m10s elapsed]
2024-05-23T15:00:10.2317191Z module.test_workspace.azurerm_databricks_workspace.this: Still creating... [1m20s elapsed]
2024-05-23T15:00:20.2318621Z module.test_workspace.azurerm_databricks_workspace.this: Still creating... [1m30s elapsed]
2024-05-23T15:00:30.2340782Z module.test_workspace.azurerm_databricks_workspace.this: Still creating... [1m40s elapsed]
2024-05-23T15:00:40.2344669Z module.test_workspace.azurerm_databricks_workspace.this: Still creating... [1m50s elapsed]
2024-05-23T15:00:50.2347768Z module.test_workspace.azurerm_databricks_workspace.this: Still creating... [2m0s elapsed]
2024-05-23T15:01:00.2360652Z module.test_workspace.azurerm_databricks_workspace.this: Still creating... [2m10s elapsed]
2024-05-23T15:01:10.2367974Z module.test_workspace.azurerm_databricks_workspace.this: Still creating... [2m20s elapsed]
2024-05-23T15:01:20.2370467Z module.test_workspace.azurerm_databricks_workspace.this: Still creating... [2m30s elapsed]
2024-05-23T15:01:30.2391168Z module.test_workspace.azurerm_databricks_workspace.this: Still creating... [2m40s elapsed]
2024-05-23T15:01:40.2393572Z module.test_workspace.azurerm_databricks_workspace.this: Still creating... [2m50s elapsed]
2024-05-23T15:01:41.3200630Z module.test_workspace.azurerm_databricks_workspace.this: Creation complete after 2m51s [id=/subscriptions/***/resourceGroups/rg-dev-testncc-camilo-001/providers/Microsoft.Databricks/workspaces/dbw-dev-testncc-camilo-001]
2024-05-23T15:01:41.7941914Z module.databricks_resources.databricks_mws_ncc_binding.this[0]: Creating...
2024-05-23T15:01:41.8051034Z module.databricks_resources.databricks_metastore_assignment.this: Creating...
2024-05-23T15:01:41.8070172Z module.databricks_resources.databricks_sql_endpoint.this: Creating...
2024-05-23T15:01:46.7895630Z module.databricks_resources.databricks_metastore_assignment.this: Creation complete after 5s [id=3742926208688921|bb749f25-4151-4698-8da3-e6ad22ac77af]
2024-05-23T15:01:46.7939865Z module.databricks_resources.databricks_storage_credential.this: Creating...
2024-05-23T15:01:46.7941100Z module.databricks_resources.databricks_grant.service_principal: Creating...
2024-05-23T15:01:47.6628574Z module.databricks_resources.databricks_mws_ncc_binding.this[0]: Creation complete after 6s [id=3742926208688921/89808a77-c403-43e0-86bb-3e0380f9c3f1]
2024-05-23T15:01:51.8081897Z module.databricks_resources.databricks_sql_endpoint.this: Still creating... [10s elapsed]
2024-05-23T15:01:51.9973537Z module.databricks_resources.databricks_storage_credential.this: Creation complete after 5s [id=dbac-testncc-camilo-001]
2024-05-23T15:01:52.4730552Z module.databricks_resources.databricks_grant.service_principal: Creation complete after 5s [id=metastore/bb749f25-4151-4698-8da3-e6ad22ac77af/***]
2024-05-23T15:01:52.4753248Z module.databricks_resources.databricks_external_location.this["catalog_001"]: Creating...
2024-05-23T15:01:53.9112331Z module.databricks_resources.databricks_external_location.this["catalog_001"]: Creation complete after 2s [id=sttestncccamilo_00_catalog]
2024-05-23T15:01:59.6473185Z module.databricks_resources.databricks_sql_endpoint.this: Creation complete after 18s [id=04d84a8333b4e634]
2024-05-23T15:01:59.7189842Z ╷
2024-05-23T15:01:59.7190553Z │ Error: Provider produced inconsistent final plan
2024-05-23T15:01:59.7190823Z │ 
2024-05-23T15:01:59.7191066Z │ When expanding the plan for
2024-05-23T15:01:59.7191429Z │ module.databricks_resources.databricks_catalog.this["catalog_001"] to
2024-05-23T15:01:59.7191801Z │ include new values learned so far during apply, provider
2024-05-23T15:01:59.7192172Z │ "registry.terraform.io/databricks/databricks" produced an invalid new value
2024-05-23T15:01:59.7192554Z │ for .storage_root: was
2024-05-23T15:01:59.7192910Z │ cty.StringVal("abfss://[email protected]"),
2024-05-23T15:01:59.7193183Z │ but now
2024-05-23T15:01:59.7193513Z │ cty.StringVal("abfss://[email protected]/").
2024-05-23T15:01:59.7193773Z │ 
2024-05-23T15:01:59.7194100Z │ This is a bug in the provider, which should be reported in the provider's
2024-05-23T15:01:59.7194369Z │ own issue tracker.
2024-05-23T15:01:59.7194584Z ╵

A subsequent terraform apply run manages to correctly propagate the external location's url.

Steps to Reproduce

1. Fill the necessary input variables.
2. terraform apply

Terraform and provider versions

Is it a regression?

Debug Output

See error message above.

Important Factoids

Would you like to implement a fix?

Yes (with some support)