storage-credential: creation on azure utilizing a service principle fails

[hargut]

Describe the issue

Using a DATABRICKS_TOKEN which was issued with az cli and trying to create a storage-credential fails with:

databricks --debug storage-credentials create --json '<json_content>'
time=2023-12-19T15:47:25.253Z level=INFO source="root.go 55}" msg=start pid=909 version=0.210.2 args="cli/databricks, --debug, storage-credentials, create, --json, {\"name\":\"<identity_name>\",\"read_only\":false,\"azure_managed_identity\":{\"access_connector_id\":\"/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.Databricks/accessConnectors/<connector_name>\",\"managed_identity_id\":\"/subscriptions/<subscription_id>/resourcegroups/<resource_group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<identity_name>\"}}"
time=2023-12-19T15:47:25.853Z level=DEBUG source="api_client.go 218}" msg="non-retriable error: Refresh token not found for userId: Some(<number>)" pid=909 sdk=true
time=2023-12-19T15:47:25.854Z level=DEBUG source="api_client.go 326}" msg="POST /api/2.1/unity-catalog/storage-credentials\n> {\n>   \"azure_managed_identity\": {\n>     \"access_connector_id\": \"/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Mi... (69 more bytes)\",\n>     \"managed_identity_id\": \"/subscriptions/<subscription_id>/resourcegroups/<resource_group>/prov... (84 more bytes)\"\n>   },\n>   \"name\": \"<identity_name>\",\n>   \"read_only\": false\n> }\n< HTTP/2.0 404 Not Found\n< {\n<   \"details\": [\n<     {\n<       \"@type\": \"type.googleapis.com/google.rpc.RequestInfo\",\n<       \"request_id\": \"93946d5f-3041-4972-a777-0d7266480f6a\",\n<       \"serving_data\": \"\"\n<     }\n<   ],\n<   \"error_code\": \"RESOURCE_DOES_NOT_EXIST\",\n<   \"message\": \"Refresh token not found for userId: Some(<number>)\"\n< }" pid=909 sdk=true
Error: Refresh token not found for userId: Some(<number>)

The service principal has the according permissions on the account and metastore.

Seems to be a similar situation with terraform & service principal credentials: https://github.com/databricks/terraform-provider-databricks/issues/3022

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

1. Run az login using a service principal
2. Obtain a token for databricks with the service principal
3. Run databricks --debug storage-credentials create --json '<json_content>' trying to create a storage-credential

Expected Behavior

Storage credential should be created successfully.

Actual Behavior

command issued a not found error

OS and CLI version

Databricks CLI v0.210.2 Linux

[hargut]

https://github.com/databricks/cli/issues/1108#issuecomment-1905946370

[nkvuong]

This likely has the same root cause as https://github.com/databricks/terraform-provider-databricks/issues/2828#issuecomment-1966424366

[antsok]

@andrewnester can you please explain why it was decided not to fix this defect?

[andrewnester]

@antsok it seems to be fixed on the latest version of CLI, do you still experience it?