[ISSUE] Issue with `databricks_grant` resource

[syvanpera]

trafficstars

Configuration

resource "databricks_grant" "this" {
  schema = "some_schema"

  principal  = "some_principal"
  privileges = ["USE_SCHEMA"]
}

Expected Behavior

For the schema grant to be created and the terraform validation to pass.

Actual Behavior

Terraform gives the following error:

Error: cannot create grant: permissions for schema-xxxx are &{[{[some_principal] [USE_SCHEMA] [Principal]}]}, but have to be {[{[some_principal [USE_SCHEMA] []}]}

Important Factoids

Went through the source for both the Go SDK and this provider and it seems this is because of the field ForceSendFields in PrivilegeAssignment struct. The provider creates the reference struct using default values and this field ends up empty and it is compared to the actual value fetched via the API using Go SDK, which sets this field to Principal (not sure where this happens, as I didn't have time yet to dig deeper).

The ForceSendFields is not part of the API response so it shouldn't be used when comparing the response and the reference object.

[syvanpera]

Note that the provider is actually using the diff function from Go SDK to compare the reference and actual object so this might also be an issue in the Go SDK and not the provider itself.

[mgyucht]

Thanks @syvanpera, this has been on my backlog to investigate and fix for a long time, but it's been a bit lower on my backlog. I'm taking another look at this today. I'll be honest, I'm not able to easily reproduce this issue. I'm using the following configuration:

resource "databricks_catalog" "example_catalog" {
  name = "example_catalog"
}

resource "databricks_schema" "example_schema" {
  catalog_name = databricks_catalog.example_catalog.name
  name         = "example_schema"
}

resource "databricks_service_principal" "example_sp" {
  display_name = "sp"
}

resource "databricks_grant" "example_permissions" {
  schema = "${databricks_catalog.example_catalog.name}.${databricks_schema.example_schema.name}"

  principal = databricks_service_principal.example_sp.application_id
  privileges = ["USE_SCHEMA", "CREATE_VOLUME"]
}

No matter what changes I make to the privileges, the apply succeeds, and the updates are propagated immediately. I'll try out with more schemas to see if it is a load-related issue.

[mgyucht]

Is it possible that you have debug logs from a run of Terraform apply with this behavior? If I can see the actual traffic between the provider and API, that will give me much more information about what is going wrong.

[syvanpera]

@mgyucht thank you very much for taking a look at this. And I'm very sorry it took me this long to get back to this. For some reason I did not receive a notification about changes to this issue and I only noticed this recently when I got back to this feature and remembered that I have this issue open.

I just tried this again and if I use another principal for this it seems to work. The one I used originally did not. I'm guessing this might have something to do if the principal already has some grants to that schema. I will need to check and come back to this. But it might be the original issue I had is actually not quite right

[leon-acce]

Hi, @syvanpera and @mgyucht I have the same error message for schema grant and I can confirm your observation - this seems to happen when I created the grant on the first apply, and on further applies, change the grant for a principal.

I can trigger the error when I do the following. But it does not seem to be consistent.

# First run
resource "databricks_grant" "schemas" {
  for_each   = toset(var.schemas)
  schema     = "${databricks_catalog.this.id}.${each.value}"
  principal  = "${var.prefix}_data-manager"
  privileges = ["MANAGE"]

  depends_on = [
    databricks_grants.catalog_reset,
    databricks_schema.this
  ]
}

# Second run, the same but change
privileges = ["MANAGE", "ALL_PRIVILEGES"]

# Third run, the same but change back to
privileges = ["MANAGE"]

Leading to error

│ Error: cannot update grant: permissions for schema-platform_central_prod.curated are &{[]}, but have to be {[{platform-central-prod_data-manager [MANAGE] []}]}
│ 
│   with module.metastore_assets[0].module.catalog.module.catalog["platform_central_prod"].databricks_grant.schemas["curated"],
│   on ../modules/catalog/catalog/main.tf line 62, in resource "databricks_grant" "schemas":
│   62: resource "databricks_grant" "schemas" {

[MHenn1g]

For us this also happens on configuration drift, when adding ALL_PRIVILEGES through the Databricks ui.

So, when the grant is created as follows.

resource "databricks_grant" "external_location" {
  external_location = "<LOCATION_NAME>"
  principal         = "<PRINCIPAL>"
  privileges        = ["READ_FILES", "BROWSE"]
}

Then somebody manually adds ALL_PRIVILEGES through the Databricks UI. The next plan looks as follows:

 ~ resource "databricks_grant" "external_location" {
        id                = "<ID>"
      ~ privileges        = [
          - "ALL_PRIVILEGES",
          - "READ_FILES",
          + "READ FILES",
            # (1 unchanged element hidden)
        ]
        # (2 unchanged attributes hidden)
    }

Which is correct. However the subsequent apply fails as mentioned before:

│ Error: cannot update grant: permissions for <RESOURCE_NAME> are [], but have to be [{<PRINCIPAL> [READ_FILES BROWSE] []}]

Despite the documentation mentioning:

Terraform will handle any configuration drift for the specified principal on every terraform apply run, even when grants are changed outside of Terraform state.

To note:

• it only happens if ALL_PRIVILEGES is added
• the latest provider 1.84.0 is used