Cannot configure default credentials in Gitlab pipelines

[ozw1z5rd]

trafficstars

Describe the issue

I don't know if this is a "classic" bug, or a improvement request. The fact is that databricks-cli fails to recover the token from the configuration file when used inside a Gitlab pipeline. Anything works until you try to use the token. BTW, the solution is don't initialize the databricks-cli, just pass TOKEN and WORKSPACE URL as env VARS.

Steps to reproduce the behavior

Of course, you need a Gitlab pipeline.

0. obtain a person token from databricks
1. In the Gitlab's repo settings -> CI/CD -> variables -> add a new VAR "TF_VAR_personal_token" to hold your PERSONAL TOKEN and set it Masked & Expanded. ( actuall, any var name is fine, but DATABRICKS_TOKEN )
2. Use a YAML file like that where the databricks cli is initialized by writing the configuration file:

variables: 
  DATABRICKS_HOST: "<your workspace url>"

stages: 
  - deploy

deploy-job:    
  stage: deploy 
  stage: build
  environment: production
  script:
    # install databricks cli 
    - curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
    - echo $TF_VAR_personal_token | databricks configure 
    - databricks current-user  me

Expected Behavior

The Gitlab pipeline ends reporting the current user info

Actual Behavior

Anything is perfect until databricks current-user me is executed. You will get this message:

https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication
to configure credentials for your preferred authentication method. Config: host=
<your workspace url>. Env: DATABRICKS_HOST

OS and CLI version

Linux, Databricks CLI V0.213.0

Debug Logs

$ if [ -z $TF_VAR_personal_token ]; then echo "Zero size TF_VAR_personal_token"; else echo $TF_VAR_personal_token | databricks configure $DEBUG; fi
12:06:19  INFO start pid=292 version=0.214.0 args="databricks, configure, --debug"
12:06:19  INFO Saving /root/.databrickscfg pid=292
12:06:19  INFO completed execution pid=292 exit_code=0
$ echo "DATABRICKS_HOST=$DATABRICKS_HOST, DATABRICKS_PATH=$DATABRICKS_PATH, token md5 will follow"
DATABRICKS_HOST=https://xxxxxx.cloud.databricks.com/, DATABRICKS_PATH=/Volumes/main/default/datalake, token md5 will follow
$ echo $TF_VAR_personal_token | md5sum
2b7425a1a31e169cce7e149de25635f8  - THIS IS CORRECT, THE TOKEN IS CORRECTLY PASSED ...
$ id
uid=0(root) gid=0(root) groups=0(root)
$ cat /root/.databrickscfg | sed s/$TF_VAR_personal_token/xxxxxxxxxxxxx/g
[DEFAULT]
host  = https://xxxxxx.cloud.databricks.com/
token = xxxxxxxxxxxxx ---- ... AND STORED
$ databricks auth env | sed s/$TF_VAR_personal_token/xxxxxxxxxxxxx/g
{
  "env": {
    "DATABRICKS_AUTH_TYPE": "pat",
    "DATABRICKS_CONFIG_PROFILE": "DEFAULT",
    "DATABRICKS_HOST": "https://xxxxxx.cloud.databricks.com/",
    "DATABRICKS_TOKEN": "xxxxxxxxxxxxx"
  }
} 
$ databricks current-user  me $DEBUG
12:06:20  INFO start pid=310 version=0.214.0 args="databricks, current-user, me, --debug"
Error: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method. Config: host=https://xxxxxx.cloud.databricks.com/. Env: DATABRICKS_HOST
12:06:20 ERROR failed execution pid=310 exit_code=1 error="default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method. Config: host=https://xxxxxx.cloud.databricks.com/. Env: DATABRICKS_HOST"

[andrewnester]

Could you try to upgrade to 0.217.1 version and run databricks auth describe command and share the output?