[ISSUE] Issue with `databricks_grant` resource. Resources get created correctly but terraform errors.

[luukschagen]

When provisioning databricks_grant resources (and I believe the same thing occurs on databricks_grants (plural) resources, though this is less extensively tested), the resources seem to get created fine on Databricks, but the provider appears to keep looping and waiting for the update, until it reports with an error that the grants don't match. Subsequent runs then error with a different error, saying that the grants are duplicate.

Configuration

terraform {
  required_providers {
    databricks = {
      source = "databricks/databricks"
    }
  }
}
 
locals {
  workspace_id = <workspace_id>
  metastore_id = <metastore_id>
}
 
provider "databricks" {
 
}
 
resource "databricks_catalog" "test" {
  metastore_id = databricks_metastore_assignment.primary.metastore_id
  name         = "testcatalog"
}
 
resource "databricks_metastore_assignment" "primary" {
  workspace_id = local.workspace_id
  metastore_id = local.metastore_id
}
 
resource "databricks_grant" "test_all" {
  principal  = "account users"
  privileges = ["ALL PRIVILEGES"]
  catalog    = databricks_catalog.test.id
}

Expected Behavior

After running the above minimal snippet, I expect to see the metastore connected to the workspace, with one created catalog called 'testcatalog', and the group 'account users' having ALL PRIVILEGES on this catalog, with terraform exiting successfully.

Actual Behavior

In reality, all of the abovementioned happens correctly, except that terraform errors out with the following error message.

Error: cannot create grant: permissions for catalog-testcatalog are &{[{account users [ALL_PRIVILEGES] [Principal]}]}, but have to be {[{account users [ALL PRIVILEGES] []}]}
│
│   with databricks_grant.test_all,
│   on main.tf line 24, in resource "databricks_grant" "test_all":
│   24: resource "databricks_grant" "test_all" {
│
╵

The actual resources on databricks are created correctly. But it appears that the 'diff' check between the existing and desired resources erroneously reports that the states don't match, even though they do: https://github.com/databricks/terraform-provider-databricks/blob/a53018ed83b7aec1a27703c8c40be8b222b346ba/catalog/permissions/permissions.go#L63

Subsequent runs therefore fail again, but now with a different error, reporting that the grants already exist

Steps to Reproduce

1. Make sure a databricks workspace and UC metastore are already present, though not yet necessarily linked
2. Fill locals <workspace_id> and <metastore_id> with appropriate values
3. terraform init & terraform apply

Terraform and provider versions

Terraform v1.7.4 on windows_amd64

databricks/databricks/1.37.1/windows_amd64

Is it a regression?

Don't know

Debug Output

Debug Logs

2024-02-29T14:06:12.080+0100 [INFO]  Starting apply for databricks_grant.test_all
2024-02-29T14:06:12.081+0100 [DEBUG] databricks_grant.test_all: applying the planned Create change
2024-02-29T14:06:12.380+0100 [DEBUG] provider.terraform-provider-databricks_v1.37.1.exe: GET /api/2.1/unity-catalog/permissions/catalog/testcatalog
< HTTP/2.0 200 OK
< {}: @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_id=3af255ed-7cea-2e78-4b9e-9886ab3673e8 tf_resource_type=databricks_grant @module=databricks tf_rpc=ApplyResourceChange timestamp="2024-02-29T14:06:12.379+0100"
2024-02-29T14:06:12.695+0100 [DEBUG] provider.terraform-provider-databricks_v1.37.1.exe: PATCH /api/2.1/unity-catalog/permissions/catalog/testcatalog
> {
>   "changes": [
>     {
>       "add": [
>         "ALL PRIVILEGES"
>       ],
>       "principal": "account users"
>     }
>   ]
> }
< HTTP/2.0 200 OK
< {
<   "privilege_assignments": [
<     {
<       "principal": "account users",
<       "privileges": [
<         "ALL_PRIVILEGES"
<       ]
<     }
<   ]
< }: tf_req_id=3af255ed-7cea-2e78-4b9e-9886ab3673e8 @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 @module=databricks tf_provider_addr=registry.terraform.io/databricks/databricks tf_resource_type=databricks_grant tf_rpc=ApplyResourceChange timestamp="2024-02-29T14:06:12.695+0100"
2024-02-29T14:06:12.695+0100 [DEBUG] provider.terraform-provider-databricks_v1.37.1.exe: Waiting for state to become: [success]: timestamp="2024-02-29T14:06:12.695+0100"
2024-02-29T14:06:12.995+0100 [DEBUG] provider.terraform-provider-databricks_v1.37.1.exe: GET /api/2.1/unity-catalog/permissions/catalog/testcatalog
< HTTP/2.0 200 OK
< {
<   "privilege_assignments": [
<     {
<       "principal": "account users",
<       "privileges": [
<         "ALL_PRIVILEGES"
<       ]
<     }
<   ]
< }: tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_id=3af255ed-7cea-2e78-4b9e-9886ab3673e8 tf_resource_type=databricks_grant tf_rpc=ApplyResourceChange @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 @module=databricks timestamp="2024-02-29T14:06:12.994+0100"
2024-02-29T14:06:12.995+0100 [DEBUG] provider.terraform-provider-databricks_v1.37.1.exe: Permissions for catalog-testcatalog are: &{[{account users [ALL_PRIVILEGES] [Principal]}]}: timestamp="2024-02-29T14:06:12.994+0100"
2024-02-29T14:06:13.813+0100 [DEBUG] provider.terraform-provider-databricks_v1.37.1.exe: GET /api/2.1/unity-catalog/permissions/catalog/testcatalog
< HTTP/2.0 200 OK
< {
<   "privilege_assignments": [
<     {
<       "principal": "account users",
<       "privileges": [
<         "ALL_PRIVILEGES"
<       ]
<     }
<   ]
< }: tf_provider_addr=registry.terraform.io/databricks/databricks tf_rpc=ApplyResourceChange @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 @module=databricks tf_resource_type=databricks_grant tf_req_id=3af255ed-7cea-2e78-4b9e-9886ab3673e8 timestamp="2024-02-29T14:06:13.813+0100"
2024-02-29T14:06:13.813+0100 [DEBUG] provider.terraform-provider-databricks_v1.37.1.exe: Permissions for catalog-testcatalog are: &{[{account users [ALL_PRIVILEGES] [Principal]}]}: timestamp="2024-02-29T14:06:13.813+0100"

<The above is repeated a couple more times....>

2024-02-29T14:07:12.703+0100 [WARN]  provider.terraform-provider-databricks_v1.37.1.exe: WaitForState timeout after 1m0s: timestamp="2024-02-29T14:07:12.703+0100"
2024-02-29T14:07:12.703+0100 [WARN]  provider.terraform-provider-databricks_v1.37.1.exe: WaitForState starting 30s refresh grace period: timestamp="2024-02-29T14:07:12.703+0100"
2024-02-29T14:07:12.703+0100 [ERROR] provider.terraform-provider-databricks_v1.37.1.exe: Response contains error diagnostic: @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/diag/diagnostics.go:62 diagnostic_detail="" diagnostic_severity=ERROR tf_resource_type=databricks_grant diagnostic_summary="cannot create grant: permissions for catalog-testcatalog are &{[{account users [ALL_PRIVILEGES] [Principal]}]}, but have to be {[{account users [ALL PRIVILEGES] []}]}" tf_req_id=3af255ed-7cea-2e78-4b9e-9886ab3673e8 @module=sdk.proto tf_proto_version=5.4 tf_provider_addr=registry.terraform.io/databricks/databricks tf_rpc=ApplyResourceChange timestamp="2024-02-29T14:07:12.703+0100"
2024-02-29T14:07:12.714+0100 [ERROR] vertex "databricks_grant.test_all" error: cannot create grant: permissions for catalog-testcatalog are &{[{account users [ALL_PRIVILEGES] [Principal]}]}, but have to be {[{account users [ALL PRIVILEGES] []}]}
2024-02-29T14:07:12.722+0100 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-02-29T14:07:12.767+0100 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/databricks/databricks/1.37.1/windows_amd64/terraform-provider-databricks_v1.37.1.exe pid=17396
2024-02-29T14:07:12.768+0100 [DEBUG] provider: plugin exited

Important Factoids

Would you like to implement a fix?

Not proficient enough in go to contribute to a fix.

[jjesky]

I am facing the same issue. Following.

[luukschagen]

Turns out the issue above is caused by the configuration using spaces instead of underscores in the privileges. Indeed the tf provider documentation correctly states that it should be with underscores, but the 'official' names of the privileges from the general Databricks documentation uses spaces.

I guess the core of the issue is still in the fact that the version with spaces does technically 'work' in that it applies the grants , but just the diff check fails.

I suppose this could be fixed either by properly allowing the version without underscores as the pull request https://github.com/databricks/terraform-provider-databricks/pull/3292 aims to do. Or just by erroring properly, with a bit more explicit error message when applying a configuration with spaces.

So I'll leave the issue open for now, because the inconsistent behavior currently should probably still be considered a bug

[edwardfeng-db]

@luukschagen Thanks for raising. I'm reaching out to our internal team to see if they can update the documentation and error message to make them consistent.

[VOVELEE]

I am facing the same issue and it is quite annoying :) Adding the underscores makes the magic but still it will be great if this is fixed.

[pcsrijith]

1. Cleanup the current roles manually
2. Kindly replace "spaces" with "underscores" . For e.g. USE CATALOG should be replaced with USE_CATALOG
3. Run terraform again.

This should resolve the issue.