[ISSUE] Issue with databricks provider authentication for GCP

[pratik-pagade]

Hi there,

My provider configuration for databricks(GCP) is no longer working. When I run a terraform apply I get the following error. I'm trying to create an account level resource(databricks group) using the following provider block. The last time (almost 4 months ago when I did run the terraform apply it worked. ) I was able to create databricks_metastore resource as well. I'm using auth_type as "google-credentials".

Configuration

provider "databricks" {
  auth_type              = "google-credentials"
  alias                   = "mws"
  host                   = "https://accounts.gcp.databricks.com"
  google_service_account = google_service_account.db_account_admin_sa.email
  google_credentials     = var.db_google_credential
  account_id             = var.db_account_id
}

resource "databricks_group" "test_tf_group" {
  display_name = "test_tf_group"
}

Tf variables:
1. db_account_id = Databricks Account ID
2. db_google_credential = Json key for the Service Account which is added as Databricks Account Admin in the DB console

TF plan:

plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # databricks_group.test_tf_group will be created
  + resource "databricks_group" "test_tf_group" {
      + acl_principal_id           = (known after apply)
      + allow_cluster_create       = false
      + allow_instance_pool_create = false
      + databricks_sql_access      = false
      + display_name               = "test_tf_group"
      + id                         = (known after apply)
      + url                        = (known after apply)
      + workspace_access           = false
    }

NOTE: We also use GOOGLE_CREDENTIALS env var in terraform but its used for other resources/provider in the repo.

Expected Behavior

Resource is created

Actual Behavior

vertex "databricks_group.test_tf_group" error: cannot create group: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method. Config: google_credentials=***. Env: GOOGLE_CREDENTIALS

Steps to Reproduce

1. terraform apply

Terraform and provider versions

tf version: 1.5.3 provider: "databricks/databricks" provider version : (1.32/1.39) Initially tried with 1.32 and then with 1.39

Is it a regression?

Fails with any provider version now. But back to 4 months it used to work with 1.32 provider version

Debug Output

https://gist.github.com/pratik-pagade/b48c0ad7433d5df927969b99a62749c9

[ROM-mm]

I’m facing the same issue. As of April 2025, the Databricks Terraform Provider only accepts tokens for the Account Console (accounts.cloud.databricks.com), and client_credentials authentication only works for Workspaces, not for the Account-level APIs.

[eahrend]

Having the same issue now as well. I'm setting the provider to use the google_service_account, checked that my ADC is setup correctly, but it's still failing.

Error: cannot read token: failed during request visitor: default auth: databricks-cli: a new access token could not be retrieved because the refresh token is invalid. If using the CLI, run the following command to reauthenticate:
  $ databricks auth login --host ######

Okay, I got mine fixed, I had to add the AuthType to "google-id" in the providers.