[ISSUE] Issue with `databricks_mws_workspaces` resource with SA and oauth2

[Israphel]

Configuration

# Databricks (Workspaces management)
provider "databricks" {
  alias         = "mws"
  host          = "https://accounts.cloud.databricks.com"
  account_id    = local.databricks_account_id
  client_id     = var.databricks_client_id
  client_secret = var.databricks_client_secret
}

Expected Behavior

I was using username and password but that doesn't work anymore. So I made a service account in here and I gave it Account admin role. Then I made an OAuth secrets and I stored the secret as var.databricks_client_secret and the UUID of the SA as var.databricks_client_id

Actual Behavior

I get

│ Error: cannot read mws workspaces: cannot read token: inner token: oauth2: "invalid_client" "Client authentication failed"
│
│   with module.databricks-workspace-us-east-1.databricks_mws_workspaces.workspace,
│   on .terraform/modules/databricks-workspace-us-east-1/main.tf line 2, in resource "databricks_mws_workspaces" "workspace":
│    2: resource "databricks_mws_workspaces" "workspace" {

Steps to Reproduce

terraform plan

Terraform and provider versions

terraform {
  required_version = "1.4.6"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.59.0"
    }
    databricks = {
      source  = "databricks/databricks"
      version = "1.25.1"
    }
  }
}

Is it a regression?

It worked when username/password was accepted.

[nkvuong]

Could you upgrade the databricks provider and see if this resolves the issue? If not, you'll need to manually add the new service accounts to the workspace as a workspace admin

[Israphel]

Could you upgrade the databricks provider and see if this resolves the issue? If not, you'll need to manually add the new service accounts to the workspace as a workspace admin

I tried several, that error didn't change.

Can you elaborate about the SAs? I made the SA at account level and not workspace level, since the provider block for the resources inside the workspace are authenticated with the output of the resource called workspace.

[nkvuong]

I think the issue is that you have provided the wrong client_id. it is not the UUID of the SA, but the client ID of the OAuth secret

Then I made an OAuth secrets and I stored the secret as var.databricks_client_secret and the UUID of the SA as var.databricks_client_id

[Israphel]

I think the issue is that you have provided the wrong client_id. it is not the UUID of the SA, but the client ID of the OAuth secret

Then I made an OAuth secrets and I stored the secret as var.databricks_client_secret and the UUID of the SA as var.databricks_client_id

The documentation says to use application_id, which is the same value as UUID (it's the same for every token).

Next to each token there's something called simply "ID", not client id, that doesn't seem to have an use.

So not sure what you mean.

For the record: the client + token work correctly, I can do a data to get workspaces and it works. But it doesn't work with the same code I had to create and manage workspaces, and the whole code follows databricks e2e example.

[nkvuong]

yes, you are correct. the actual issue is a bit more nuanced: since you specified the token block in mws_workspace, the TF provider tries to authenticate with the workspace API using the account admin credential to generate a new PAT. This works for the workspace creator, but for other account admins, they have to be added to the workspace as an admin as well.

Essentially, step 1 in this doc needs to be completed - https://docs.databricks.com/en/dev-tools/auth/oauth-m2m.html#step-1-create-a-service-principal

[Israphel]

Ok, once I create SAs in the workspace, where do I use them within the code?

# Databricks (Workspaces management)
provider "databricks" {
  alias         = "mws"
  host          = "https://accounts.cloud.databricks.com"
  account_id    = local.databricks_account_id
  client_id     = var.databricks_client_id
  client_secret = var.databricks_client_secret
}

# Databricks (us-east-1)
provider "databricks" {
  alias = "use1"
  host  = module.databricks-workspace-us-east-1.url
  token = module.databricks-workspace-us-east-1.token
}

# Databricks (us-west-2)
provider "databricks" {
  alias = "usw2"
  host  = module.databricks-workspace-us-west-2.url
  token = module.databricks-workspace-us-west-2.token
}

I can't use the module's output anymore? like the e2e example says

or you're saying that: as long as the SA belongs to the workspace creator, it should work?

[radusora]

I can confirm I have a similar issue after moving from user/password auth to a service principle and oauth2. Interestingly enough, it works, but intermittently. When it fails the error is

│ Error: cannot read storage credential: failed during request visitor: inner token: Post "https://xxxxxxxxx.cloud.databricks.com/oidc/v1/token": {"error":"invalid_client","error_id":"xxx","error_description":"Client authentication failed"}

I'm running v1.49.1 of the provider.