[ISSUE] `enable_serverless_compute = false` has no effect in `databricks_sql_endpoint`

[ravirahul-db]

Have enabled the workspace for serverless SQL warehouse. All the new warehouses are created by default as serverless even when I have "enable_serverless_compute = false"

Configuration

resource "databricks_sql_endpoint" "this" {
  name                      = "ravi-tf-test"
  cluster_size              = "Small"
  max_num_clusters          = 1
  enable_serverless_compute = false
}

Expected Behavior

Provider should consider the value of "enable_serverless_compute" when creating/updating the warehouse.

Actual Behavior

"enable_serverless_compute" has no effect.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply

Terraform and provider versions

1.0.0

Debug Output

Could not see the enable_serverless_compute passed to REST api. Same applicable for edit endponit. During warehouse creation:

2022-07-08T01:03:34.759+0530 [INFO]  provider.terraform-provider-databricks: Configured pat auth: host=https://kavya.cloud.databricks.com/, token=***REDACTED***: timestamp=2022-07-08T01:03:34.759+0530
2022-07-08T01:03:34.759+0530 [DEBUG] provider.terraform-provider-databricks: POST /api/2.0/sql/endpoints {
  "auto_stop_mins": 120,
  "cluster_size": "Small",
  "enable_photon": true,
  "max_num_clusters": 1,
  "min_num_clusters": 1,
  "name": "ravi-tf-test",
  "num_clusters": 1,
  "spot_instance_policy": "COST_OPTIMIZED"
}: timestamp=2022-07-08T01:03:34.759+0530
2022-07-08T01:03:36.328+0530 [DEBUG] provider.terraform-provider-databricks: 200 OK  {
  "id": "0a8161fe7886e0cf"
} <- POST /api/2.0/sql/endpoints: timestamp=2022-07-08T01:03:36.328+0530
2022-07-08T01:03:36.329+0530 [DEBUG] provider.terraform-provider-databricks: Waiting for state to become: [success]: timestamp=2022-07-08T01:03:36.328+0530
2022-07-08T01:03:36.329+0530 [DEBUG] provider.terraform-provider-databricks: GET /api/2.0/sql/endpoints/0a8161fe7886e0cf: timestamp=2022-07-08T01:03:36.328+0530
2022-07-08T01:03:36.678+0530 [DEBUG] provider.terraform-provider-databricks: 200 OK  {
  "auto_resume": true,
  "auto_stop_mins": 120,
  "cluster_size": "Small",
  "creator_id": 6489611687929196,
  "creator_name": "[email protected]",
  "enable_photon": true,
  "enable_serverless_compute": true,
  "health": {
    "status": "HEALTHY"
  },
  "id": "0a8161fe7886e0cf",
  "jdbc_url": "jdbc:spark://kavya.cloud.databricks.com:443/default;transportMode=http;ssl=1;AuthMech=3;httpPath... (37 more bytes)",
  "max_num_clusters": 1,
  "min_num_clusters": 1,
  "name": "ravi-tf-test",
  "num_active_sessions": 0,
  "num_clusters": 0,
  "odbc_params": {
    "hostname": "kavya.cloud.databricks.com",
    "path": "/sql/1.0/endpoints/0a8161fe7886e0cf",
    "port": 443,
    "protocol": "https"
  },
  "size": "SMALL",
  "spot_instance_policy": "COST_OPTIMIZED",
  "state": "STARTING",
  "tags": {}
} <- GET /api/2.0/sql/endpoints/0a8161fe7886e0cf: timestamp=2022-07-08T01:03:36.678+0530
2022-07-08T01:03:36.678+0530 [INFO]  provider.terraform-provider-databricks: endpoint 0a8161fe7886e0cf is STARTING: timestamp=2022-07-08T01:03:36.678+0530

When updating the warehouse:

2022-07-08T00:43:13.854+0530 [DEBUG] provider.terraform-provider-databricks_v1.0.0: POST /api/2.0/sql/endpoints/31a3fd747f989319/edit {
  "auto_stop_mins": 120,
  "cluster_size": "Small",
  "data_source_id": "dfb963f7-8d92-424e-ada8-74808382732a",
  "enable_photon": true,
  "id": "31a3fd747f989319",
  "jdbc_url": "jdbc:spark://kavya.cloud.databricks.com:443/default;transportMode=http;ssl=1;AuthMech=3;httpPath... (37 more bytes)",
  "max_num_clusters": 1,
  "min_num_clusters": 1,
  "name": "ravi-tf-test",
  "num_clusters": 1,
  "odbc_params": {
    "hostname": "kavya.cloud.databricks.com",
    "path": "/sql/1.0/endpoints/31a3fd747f989319",
    "port": 443,
    "protocol": "https"
  },
  "spot_instance_policy": "COST_OPTIMIZED",
  "state": "STOPPED"
}: timestamp=2022-07-08T00:43:13.854+0530
2022-07-08T00:43:15.297+0530 [DEBUG] provider.terraform-provider-databricks_v1.0.0: 200 OK  {} <- POST /api/2.0/sql/endpoints/31a3fd747f989319/edit: timestamp=2022-07-08T00:43:15.297+0530

[alexott]

It wasn’t fixed yet - I need to update docs to describe behavior

[amirhalatzi]

Hey @alexott and @nfx ! I don't follow: was this issue resolved? This still occurs in version 1.2.1

[alexott]

No, it's not fixed yet because of the problems with the underlying APIs that were fixed only recently

[sdebruyn]

Right now all warehouses are created as serverless

[shreyas-goenka]

Solving this bug will require fundamental changes to the terraform client design to detect an unspecified enable_serverless_compute field in this specific case. These changes are expected to take a couple of months after which we can revisit this issue

[sdebruyn]

So right now there is no way to create a non-serverless SQL warehouse using this provider?

[shreyas-goenka]

This provider will create non-serverless SQL warehouses on workspaces with serverless disabled. But yes, unfortunately in workspaces where serverless is enabled, unfortunately, there is no way right now to create classic SQL warehouses.

This is because we are migrating this provider to the databricks golang SDK (https://github.com/databricks/databricks-sdk-go) and a quick fix without solving the fundamental issue would either be incompatible with this migration or would break other expected behavior

[shannon-barrow]

Any way to get a status on this? Chick-Fil-A is reporting the same issue as a blocker.

[shannon-barrow]

Solving this bug will require fundamental changes to the terraform client design to detect an unspecified enable_serverless_compute field in this specific case. These changes are expected to take a couple of months after which we can revisit this issue

@shreyas-goenka Can we get an update on this? The latest message specified it would take a couple of months then can revisit but this message was from 10 months ago. This has become a blocker for my customer to adopt serverless DBSQL now (since all infra changes happen via Terraform and not the UI)