[ISSUE] Use `dbutils.fs.updateMount` to re-mount Azure Storage after rotating SPN secret

[sdebruyn]

Terraform Version

Terraform 0.14.6 Databricks provider: any version (v0.3.0 or older)

Affected Resource(s)

Please list the resources as a list, for example:

• databricks_secret
• databricks_azure_adls_gen2_mount
• possibly all mount types

Terraform Configuration Files

resource "databricks_secret_scope" "mounts" {
  name                     = "mounts"
  initial_manage_principal = "users"
}

resource "databricks_secret" "sp_secret" {
  key          = "client_secret"
  string_value = module.service_principal.client_secret
  scope        = databricks_secret_scope.mounts.name
}

resource "databricks_azure_adls_gen2_mount" "dl" {
  depends_on             = [azurerm_role_assignment.dl_data_owner]
  for_each               = module.datalake.data_lake_layers
  container_name         = each.value
  storage_account_name   = module.datalake.storage_account_name
  mount_name             = each.value
  tenant_id              = module.service_principal.tenant_id
  client_id              = module.service_principal.client_id
  client_secret_scope    = databricks_secret_scope.mounts.name
  client_secret_key      = databricks_secret.sp_secret.key
  cluster_id             = databricks_cluster.terraform.id
  initialize_file_system = true
}

Description

When you change the value of the secret somewhere in Terraform (e.g. a new value for random_password somewhere), the secret is updated accordingly on Databricks.

However, the mount itself becomes broken. It does not seem to refresh the secret value and returns authentication errors when you try to use it within Databricks.

[nfx]

@sdebruyn for now, please re-create the mounts once you rotate the secrets, as the workaround.

~~It should be fixed in scope of a more holistic approach of #497 . Closing this issue to keep contributor team focused.~~

[nfx]

https://docs.microsoft.com/en-us/azure/databricks/dev-tools/databricks-utils#dbutils-fs-updatemount

high level implementation details:

within mount create and read commands:

1. hash secret value to sha256
2. return mount source and AND hash of secret

within terraform resource:

1. store hash of secret in terraform state after create
2. read hash of secret along with mount source every refresh
3. trigger new command, that would read new secret again and pass it to updateMount
4. trigger read command to verify that storage is readable again and refresh hash of current secret in the state

[nfx]

Good news to those interested - giving this fix a try in #1029, let's see if it'll make it to v0.4.x or v0.5.x

[w0ut0]

We're bumping in this issue when rotating our keys/access tokens. Recreating the mounts is relatively time-consuming due to https://github.com/databricks/terraform-provider-databricks/issues/687

[redcape]

Same issue here. We thought updating the mount's secrets would cause terraform to either updateMount or unmount/mount, but it does neither. We found out when the secrets expired and the mounts no longer worked.