[ISSUE] Issue with `databricks_catalog` resource -> Renaming catalogs does not work

[jacovg91]

Configuration

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
    }
    databricks = {
      source = "databricks/databricks"
    }
  }
}

provider "databricks" {
  alias                       = "workspace"
  host                        = data.azurerm_databricks_workspace.dbw.workspace_url
  azure_workspace_resource_id = data.azurerm_databricks_workspace.dbw.id
}

resource "databricks_external_location" "default_location" {
  provider        = databricks.workspace
  name            = "xyzdls01-container"
  url             = "abfss://[email protected]"
  owner           = "xyz"
  comment         = "xyz"
  credential_name = var.databricks_storage_credential_name

  depends_on = [
    databricks_metastore_assignment.enable_uc
  ]
}

resource "databricks_catalog" "catalog" {
  provider       = databricks.workspace
  metastore_id   = var.databricks_metastore_id
  name           = "def"
  comment        = ""
  owner          = "owner"
  storage_root   = databricks_external_location.default_location.url
  isolation_mode = "ISOLATED"
  properties = {
    purpose = "foo"
  }
  depends_on = [databricks_external_location.default_location]
}

# module.foo.databricks_catalog.catalog will be updated in-place
  ~ resource "databricks_catalog" "catalog" {
        id             = "abc"
      ~ name           = "abc" -> "def"
        # (7 unchanged attributes hidden)
    }

Expected Behavior

The catalog should have been renamed from abc to def

Actual Behavior

Getting the following

Error: cannot update catalog: CATALOG 'abc' does not exist

Steps to Reproduce

1. Deploy a catalog
2. Change a catalogs name
3. terraform apply

Terraform and provider versions

Terraform and provider versions

Terraform v1.5.4 on windows_amd64

Debug Output

Important Factoids

[nkvuong]

currently the Go SDK does not support renaming a catalog, as the name parameter is overloaded. There is ongoing work to standardise this for all Unity Catalog securables

[WP-LKL]

Any update here or link to where the supposed standardization work is occurring?

[mgyucht]

The underlying changes to the Go SDK are completed (see https://github.com/databricks/databricks-sdk-go/releases/tag/v0.27.0). However, to make these changes safely in TF requires a bit more work. Because in UC the name is the identifier for UC resources, renaming resources creates a race condition. For example: if a template contains a catalog A and schema C, and catalog A is renamed to B, when making a request to update schema C, do you use A.C or B.C as the "full name" of the schema when making the request? It depends on whether the catalog was updated before or after the schema during the apply.

Addressing this is a bigger challenge. We're working with the UC team to see if APIs with stable identifiers for UC resources can be exposed. Without this, we can revisit the UC APIs in TF to ensure that there is a dependency between UC resources with a hierarchy, so that updates for dependent resources can assume that their upstream dependencies are always updated first.

[stevenayers-bge]

A workaround for this is using the terraform_data built-in resource and the lifecycle argument replace_triggered_by.

It will not trigger the first time you create the terraform_data resource, but if you create the resource, then change the name of your catalog(s), it should work.

ref: https://developer.hashicorp.com/terraform/language/resources/terraform-data#example-usage-data-for-replace_triggered_by

locals {
  catalog_names = ["bronze", "silver", "gold"]
}

resource "terraform_data" "trigger_catalog_recreate" {
  for_each = toset(local.catalog_names)
  input = each.key
}

resource "databricks_catalog" "catalog" {
  provider        = databricks.workspace
  for_each        = toset(local.catalog_names)
  isolation_mode  = "ISOLATED"
  name            = each.key

  lifecycle {
    replace_triggered_by = [
      terraform_data.trigger_catalog_recreate[each.key]
    ]
  }
}