databricks-cli
databricks-cli copied to clipboard
databricks
Authentication is temporarily unavailable when creating Azure keyvault backed secret scope
We get the following message when creating a secret scope with an Azure keyvault back end in the CI.
{ 'error_code': 'TEMPORARILY_UNAVAILABLE',
'message': 'Authentication is temporarily unavailable. Please try again '
'later.'}
Locally it works, the aad-token is set in the CI script, I have tried it at different times. Anyone can help with what to do with this message?
Hi,
we observed a same issue. Its connected to aad-token authorization. I am able to connect and list secret scopes but not to create a new one.
We tried to call Databricks API directly but it is same there (so it is not problem of CLI). We are not able to deploy ntbs or create secret scope with AAD token.
@michalmlaka : Thanks for adding that it is an API thing. We'll see if they fix it there.
This currently does not work with AAD tokens acquired with Service principals
“Service Principals cannot be used with the AKV secret integration. This is because they do not support the on-behalf-of flow. We are requesting this of the AAD team and working on documenting it. You need to use a user's AAD token.”
In the interim, use below approach to create secret scope using user's AAD token
Obtain token using user creds
from adal import AuthenticationContext
authority_host_url = "https://login.microsoftonline.com/"
# the Application ID of AzureDatabricks
azure_databricks_resource_id = "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d"
# Required user input
user_parameters = {
"tenant" : "fa5cf446-0f25-4992-9e15-8723c0a0b170",
"client_id" : "5662111a-38a8-403c-9d9a-f9b690985796",
"username" : "user_upn",
"password" : "xxxxxxxxxxxxxxxxx"
}
# configure AuthenticationContext
# authority URL and tenant ID are used
authority_url = authority_host_url + user_parameters['tenant']
context = AuthenticationContext(authority_url)
# API call to get the token
token_response = context.acquire_token_with_username_password(
azure_databricks_resource_id,
user_parameters['username'],
user_parameters['password'],
user_parameters['client_id']
)
access_token = token_response['accessToken']
refresh_token = token_response['refreshToken']
access_token
REST API request to create secret scope
POST https://adb-6187942982036570.10.azuredatabricks.net/api/2.0/secrets/scopes/create HTTP/1.1
Authorization: Bearer <token>
# For Keyvault backed scope
{
"scope": "restapiaad-keyvault-scope",
"initial_manage_principal": "users",
"backend_azure_keyvault": {
"dns_name": "https://aravish369vault.vault.azure.net/",
"resource_id": "/subscriptions/f34f9848-8b93-4cfb-9ced-0c4548a74cce/resourceGroups/aravishexternalrsgrp/providers/Microsoft.KeyVault/vaults/aravish369vault"
},
"scope_backend_type": "AZURE_KEYVAULT"
}
I am able to authenticate with AAD tokens to create a databricks personal access token, then authenticate again with the api using the databricks token to create secret scopes. I'm not sure if that works with keyvault backends, but I was running into the same issue just creating databricks backend scopes
@gramhagen, Ultimately the AAD token should be used to create the secret scopes which should work as expected and not run into issues.
Unfortunately does not working. Run into these issues:
https://forums.databricks.com/questions/50626/azure-databricks-api-20-cant-create-azure-keyvault.html
@aravish , Is there ETA to fix the issue that unable to create Azure Key Vault scope using Service Principal token? Using user AAD token is not a good solution for automation that is running on service principal. We cannot store personal password in automation.
We are automating the Azure databricks configuration, including Azure resource creation, databricks cluster creation, Azure key Vault secret scope creation with CI/CD. This issue is blocking us to automate the process.
I believe the PMs were talking to msft about enabling AKV creation with service principal tokens. @annashres might be the best person to talk to about this?
I believe the PMs were talking to msft about enabling AKV creation with service principal tokens. @AnnaShrestinan might be the best person to talk to about this?
Thanks @sushi1998 for the info. How to contact AnnaShrestinan?
We tried it once again. It looks better now. (error msg.).
{"error_code":"CUSTOMER_UNAUTHORIZED","message":"Unable to grant read/list permission to Databricks service principal to KeyVault 'https://keyvaultname.vault.azure.net/': key not found: https://graph.windows.net/"}(base)
I assigned "Directory.Read.All" permission for AAD Graph API (https://graph.windows.net/). But it is not enough. What else do I need?
I think we are waiting to hear from @AnnaShrestinan like @sushi1998 mentioned. To the best of my knowledge it does not have to do with setting the right permissions for your SP.
Not that I am aware off. What I know: databricks people are talking to Azure people. It most likely takes a while: months not weeks
+1 same issue here. Please fix it. Creating a Secret Scope using a Service Principal with AAD Token is still not working.
Same issue here, please fix as in my company the only way we can do things in higher envs is with SPN so it is not possible to config any other way!
We are working with Azure to enable the use of Service Principals for creating an AKV-backed secret scope. @hpsin from Azure AD will have updates on the timeline.
@annashres @hpsin Has there been any updates on this with timelines etc.? The alternative way is just too much rework for CI/CD processes right now, especially as I am using a regular SP for everything else in the Databricks CLI
Thanks
Any updates on this? I am also having issues trying to set up key vault backed secret scope.
+1 only manual step in a huge deployment currently. Any updates on this? @hpsin
We don't have timelines to announce right now on when SP impersonation is going to be supported. We're investigating how we build this - it requires the creation of a new class of scopes (delegation scopes for service principals) and updating the authorization logic of every API. We understand the importance of this feature for automation flows, and building it securely is our number one concern.
We don't have timelines to announce right now on when SP impersonation is going to be supported. We're investigating how we build this - it requires the creation of a new class of scopes (delegation scopes for service principals) and updating the authorization logic of every API. We understand the importance of this feature for automation flows, and building it securely is our number one concern.
thanks for the update 👌
@hpsin why not just use a managed identity like azure datafactory ?? this will probably also solve some other problems related to cloud auth, currently when creating scope on UI, how the refresh token is passed to clusters ?? is the user AAD Token stored somewhere permanently ??, what happens if the user token is revoked ??
The creation of a new class of scope is almost complete, and the team is hoping to test soon. @psignoret and @annashres should be able to provide updates from here as I've transitioned away from Azure AD.
Just checking in to see how this request is tracking? Lots of users are eagerly awaiting this capability!
+1 @annashres and @psignoret, could you give the community an update regarding this issue? We are all eagerly waiting for this feature's arrival!
