[ISSUE] Issue with `databricks_access_control_rule_set` resource

[anineharto]

Configuration

variable "example_sps" {
  type = set(object({
    name                          = string
    user_group_names              = optional(set(string), [])
  }))
  description = "Azure service principal display names and user groups in Databricks"
  default     = []
}

locals {
example_sps_user_groups = flatten([
    for sp in var.example_sps : [
      for user_group_name in sp.user_group_names : {
        name  = sp.name
        group = user_group_name
      }
    ]
  ])
 }

data "azuread_service_principal" "all" {
  for_each = toset(local.all_sp_names)

  display_name = each.value
}

resource "databricks_service_principal" "example_sps" {
  for_each = {
    for sp in var.example_sps : sp.name => sp
  }

  application_id = data.azuread_service_principal.all[each.key].application_id
  display_name   = data.azuread_service_principal.all[each.key].display_name

  allow_cluster_create  = true
  databricks_sql_access = true
  workspace_access      = true
}

data "databricks_group" "user_groups" {
  for_each = {for sp in local.example_sps_user_groups : "${sp.name}:${sp.group}" => sp.group}
  display_name = each.value
}

resource "databricks_access_control_rule_set" "example_sps_users" {
  for_each = {
    for sp in var.example_sps : sp.name => sp.user_group_names
    if length(sp.user_group_names) > 0
  }
  name = "accounts/${var.databricks_account_id}/${databricks_service_principal.example_sps[each.key].acl_principal_id}/ruleSets/default"

  dynamic "grant_rules" {
    for_each = each.value
    content {
      principals = [data.databricks_group.user_groups["${each.key}:${grant_rules.value}"].acl_principal_id]
      role       = "roles/servicePrincipal.user"
    }
  }
}

Expected Behavior

According to the terraform plan the defined user groups should have user access on the databricks service principals:

databricks_access_control_rule_set.example_sps_users[<service principal name>] will be created
+ resource "databricks_access_control_rule_set" "example_sps_users" {
      + etag = (known after apply)
      + id   = (known after apply)
      + name = "accounts/<databricks account id>/servicePrincipals/<service principal application id>/ruleSets/default"

      + grant_rules {
          + principals = [
              + "groups/<user group name>",
            ]
          + role       = "roles/servicePrincipal.user"
        }
    }

Actual Behavior

Error message upon apply:

Error: cannot create access control rule set: invalid Databricks Account configuration
│ 
│   with databricks_access_control_rule_set.example_sps_users["XXX"],
│   on .terraform/xxx.tf line 42, in resource "databricks_access_control_rule_set" "example_sps_users":
│   42: resource "databricks_access_control_rule_set" "example_sps_users" {

Steps to Reproduce

1. `terraform apply

Terraform and provider versions

terraform: 1.8.1 databricks: 1.40.0

Is it a regression?

No, this is the first time we're implementing this resource.

[nkvuong]

what is your provider configuration? the error message is to do with that invalid Databricks Account configuration

[anineharto]

I'm trying to configure it on a workspace

provider "databricks" {
  host       = data.azurerm_databricks_workspace.main.workspace_url
  auth_type  = "azure-cli"
  account_id = local.databricks_account_id
}

[nkvuong]

could you remove account_id = local.databricks_account_id from the provider definition?

[anineharto]

I did, and now I get the following errors:

Error: cannot create access control rule set: Not authorized to perform this operation

Error: cannot create access control rule set: Unhandled error in API call

[brandocomando]

Having the same issue. Can set the ruleset in the account using the account provider alias, but if I try with a workspace provider I get "cannot create access control rule set: invalid Databricks Account configuration"

Additionally if I use the account level provider it sets it at the account level, not the workspace level, which is not what I'm trying to do.

[brandocomando]

So what it was for me was that I had 2 providers 1 for account, and one for workspace. But the account was loading the accountID from the environment variable. Once I unset DATABRICKS_ACCOUNT_ID and passed it into the account provider in another way it worked ok