terraform-provider-databricks

terraform-provider-databricks copied to clipboard

Starting v1.40.0, it appears that the cluster_mount_info no longer works. Explicitly defining it in the Terraform template, the configuration ends up not being sent.

Expected Behavior

It is expected that the NFS mount is present in the cluster resource

Actual Behavior

Instead, the cluster_mount_info block is being ignored. This can be seen in the debug log POST request.

Steps to Reproduce

1. terraform apply
2. Observe that there is no NFS mount present in the cluster

Terraform and provider versions

❯ terraform version
Terraform v1.7.5
on darwin_arm64
+ provider registry.terraform.io/databricks/databricks v1.40.0

Is it a regression?

Yes, it worked on v1.39.0 and below. The issue started to occur on v1.40.0+

Debug Output

# Works (v1.39.0)
2024-06-12T15:46:30.047+0000 [DEBUG] provider.terraform-provider-databricks_v1.39.0: POST /api/2.0/clusters/create
> {
>   "autotermination_minutes": 20,
>   "cluster_mount_infos": [
>     {
>       "local_mount_dir_path": "/mnt/nfs-test",
>       "network_filesystem_info": {
>         "mount_options": "sec=sys,vers=3,nolock,proto=tcp",
>         "server_address": "<REDACTED>"
>       }
>     }
>   ],
>   "cluster_name": "1.39.0",
>   "node_type_id": "m5d.large",
>   "num_workers": 2,
>   "spark_version": "14.3.x-scala2.12"
> }
< HTTP/2.0 200 OK
< {
<   "cluster_id": "0612-154629-72sqch2b"
< }: @module=databricks tf_provider_addr=registry.terraform.io/databricks/databricks tf_resource_type=databricks_cluster @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 tf_req_id=5a764950-0036-bd49-6fcc-25077d9b4ba3 tf_rpc=ApplyResourceChange timestamp="2024-06-12T15:46:30.047+0000"

# Broken (v1.40.0)
2024-06-12T15:55:32.761+0000 [DEBUG] provider.terraform-provider-databricks_v1.40.0: POST /api/2.0/clusters/create
> {
>   "autotermination_minutes": 20,
>   "cluster_name": "1.40.0",
>   "node_type_id": "m5d.large",
>   "num_workers": 2,
>   "spark_version": "14.3.x-scala2.12"
> }
< HTTP/2.0 200 OK
< {
<   "cluster_id": "0612-155532-mwtsholm"
< }: @module=databricks tf_provider_addr=registry.terraform.io/databricks/databricks tf_rpc=ApplyResourceChange @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 tf_req_id=e149de14-4a3b-b5bb-68ba-171f092c7b15 tf_resource_type=databricks_cluster timestamp="2024-06-12T15:55:32.761+0000"

# Broken (v1.47.0, latest)
2024-06-12T16:04:07.950+0000 [DEBUG] provider.terraform-provider-databricks_v1.47.0: POST /api/2.0/clusters/create
> {
>   "autotermination_minutes": 20,
>   "cluster_name": "1.47.0",
>   "node_type_id": "m5d.large",
>   "num_workers": 2,
>   "spark_version": "14.3.x-scala2.12"
> }
< HTTP/2.0 200 OK
< {
<   "cluster_id": "0612-160407-35q0w0rd"
< }: @module=databricks tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_id=2b4a9fed-38c8-be7c-da49-4303b4e7c57b tf_rpc=ApplyResourceChange @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 tf_resource_type=databricks_cluster timestamp="2024-06-12T16:04:07.950+0000"

Configuration

terraform {
  required_providers {
    databricks = {
      source = "databricks/databricks"
      version = "1.40.0"
    }
  }
}

provider "databricks" {
  profile = "<REDACTED>"
}

data "databricks_node_type" "smallest" {
  local_disk = true
}

data "databricks_spark_version" "latest_lts" {
  long_term_support = true
}

resource "databricks_cluster" "shared_autoscaling" {
  cluster_name            = "1.40.0"
  spark_version           = data.databricks_spark_version.latest_lts.id
  node_type_id            = data.databricks_node_type.smallest.id
  autotermination_minutes = 20
  num_workers = 2

  cluster_mount_info {
    network_filesystem_info {
      server_address = "<REDACTED>"
      mount_options  = "sec=sys,vers=3,nolock,proto=tcp"
    }
    remote_mount_dir_path = ""
    local_mount_dir_path  = "/mnt/nfs-test"
  }
}

Most likely caused by #3459

yes, looks like it's caused by it - missing alias or something like that

From our observation, It seems like CreateCluster struct does not haveClusterMountInfo as a field. So, this is something we need to fix on go-sdk side. cc @hectorcast-db