[ISSUE] Issue with `databricks_cluster` resource - The field 'node_type_id' cannot be supplied when an instance pool ID is provided

[prejs]

Configuration

data "databricks_spark_version" "latest_lts" {
  long_term_support = true
}

data "databricks_instance_pool" "workers" {
  name = "Pool"
}

data "databricks_instance_pool" "driver" {
  name = "Pool"
}

resource "databricks_cluster" "this" {
  cluster_name            = "Cluster"
  spark_version           = data.databricks_spark_version.latest_lts.id
  instance_pool_id        = data.databricks_instance_pool.workers.id
  driver_instance_pool_id = data.databricks_instance_pool.driver.id

  # Update works for this
  autoscale {
    min_workers = 0
    max_workers = 2
  }
  # Update fails for this
  autotermination_minutes = 10
}

Expected Behavior

Update request should not contain node_type_id nor driver_node_type_id. Cluster should be successfully updated.

Actual Behavior

Update fails.

│ Error: cannot update cluster: The field 'node_type_id' cannot be supplied when an instance pool ID is provided. │ │ with databricks_cluster.this, │ on main.tf line 13, in resource "databricks_cluster" "this": │ 13: resource "databricks_cluster" "this" {

Steps to Reproduce

1. Use existing pool's name in lines 6 and 10
2. terraform apply
3. Change min_workers = 1 in line 21
4. terraform apply
5. Change autotermination_minutes = 20 in line 25
6. terraform apply

Terraform and provider versions

OpenTofu v1.7.0
on darwin_arm64
+ provider registry.opentofu.org/databricks/databricks v1.41.0

Our pipelines use Terraform with the same results.

Terraform v1.8.1
on linux_amd64
+ provider registry.terraform.io/databricks/databricks v1.41.0

Is it a regression?

Yes, it works in v1.39.0.

Debug Output

https://gist.github.com/prejs/7c447c114d5e3cd2fb70aa039decfdb3

Important Factoids

n/a

Would you like to implement a fix?

I took a look at significant changes introduced with v1.40.0, and the only suspicious place I see is passing ClusterSpec by value to FixInstancePoolChangeIfAny function. I hope the maintainers can pick up from here, since I don't have an idea how to verify my suspicion.

[tanmay-db]

Hi @prejs, thanks for raising the issue and taking a look. I will follow up on the issue and post update soon.