Azure
az storage blob sync fails with "--auth-mode login"
Describe the bug
We are trying to use azure CLI to sync data from a local folder to an azure storage account that has Storage account key access set to Disabled. Our runner (Github Actions) uses the auth method OpenID Connect, which is a form of active directory Service principal, and is unable to obtain an explicit service principal key. We therefore require az storage blob sync to support --auth-mode login.
az storage blob upload-batch --auth-mode login does work, but is extremely slow (hours instead of minutes) as we have a large number of blobs that are the same in source and target for our sync jobs.
Related command
Any variation of az storage blob sync --auth-mode login will fail. Below is an example:
az storage blob sync --auth-mode login --account-name $storageacct --source "/repo" --container repo
Errors
unrecognized arguments: --auth-mode
Issue script & Debug output
Run az storage blob sync --auth-mode login --account-name $storageacct --source "/home/runner/work/repo" --container repo --debug
DEBUG: cli.knack.cli: Command arguments: ['storage', 'blob', 'sync', '--auth-mode', 'login', '--account-name', 'qberdlaketst', '--source', '/home/runner/work/repo', '--container', 'repo', '--debug']
DEBUG: cli.knack.cli: init debug log:
Cannot enable color.
DEBUG: cli.knack.cli: Event: Cli.PreExecute []
DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7fc73ef712d0>, <function OutputProducer.on_global_arguments at 0x7fc73ee73eb0>, <function CLIQuery.on_global_arguments at 0x7fc73eec1120>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
DEBUG: cli.azure.cli.core: Modules found from index for 'storage': ['azure.cli.command_modules.storage']
DEBUG: cli.azure.cli.core: Loading command modules:
DEBUG: cli.azure.cli.core: Name Load Time Groups Commands
DEBUG: cli.azure.cli.core: storage 0.054 58 272
DEBUG: cli.azure.cli.core: Total (1) 0.054 58 272
DEBUG: cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
DEBUG: cli.azure.cli.core: Loading extensions:
DEBUG: cli.azure.cli.core: Name Load Time Groups Commands Directory
DEBUG: cli.azure.cli.core: Total (0) 0.000 0 0
DEBUG: cli.azure.cli.core: Loaded 58 groups, 272 commands.
DEBUG: cli.azure.cli.core: Found a match in the command table.
DEBUG: cli.azure.cli.core: Raw command : storage blob sync
DEBUG: cli.azure.cli.core: Command table: storage blob sync
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7fc73dd0eb90>]
DEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/runner/.azure/commands/2023-06-21.03-59-08.storage_blob_sync.2264.log'.
INFO: az_command_data_logger: command args: storage blob sync --auth-mode {} --account-name {} --source {} --container {} --debug
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.
DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last): File "/opt/az/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 655, in _get_attr op = getattr(op, part) AttributeError: module 'azure.mgmt.storage.v2022_09_01.models' has no attribute 'ListKeyExpand'
DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last): File "/opt/az/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 655, in _get_attr op = getattr(op, part) AttributeError: module 'azure.mgmt.storage.v2022_09_01.models' has no attribute 'CorsRuleAllowedMethodsItem'
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.
ERROR: cli.azure.cli.core.azclierror: unrecognized arguments: --auth-mode ERROR: az_command_data_logger: unrecognized arguments: --auth-mode
Examples from AI knowledge base: az storage blob sync --container mycontainer --source "path/to/file" --destination NewBlob Sync a single blob to a container.
az storage blob sync --container mycontainer --account-name mystorageccount --account-key 00000000 --source "path/to/directory" Sync a directory to a container.
https://docs.microsoft.com/en-US/cli/azure/storage/blob#az_storage_blob_sync Read more about the command in reference docs DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7fc73dd0edd0>] INFO: az_command_data_logger: exit code: 2 INFO: cli.main: Command ran in 2.474 seconds (init: 0.181, invoke: 2.293) INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1 INFO: telemetry.client: Accumulated 0 events. Flush the clients. INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1 INFO: telemetry.save: Save telemetry record of length 3728 in cache WARNING: telemetry.check: Negative: The /home/runner/.azure/telemetry.txt was modified at 2023-06-21 03:58:23.812199, which in less than 600.000000 s Error: Process completed with exit code 2.
Expected behavior
Behave like az storage blob upload-batch (or other az storage blob commands) and support --auth-mode login
Environment Summary
azure-cli 2.49.0 core 2.49.0 telemetry 1.0.8 Extensions: azure-devops 0.26.0 resource-graph 2.1.0 Dependencies: msal 1.20.0 azure-mgmt-resource 22.0.0 Python location '/opt/az/bin/python3' Extensions directory '/opt/az/azcliextensions' Python (Linux) 3.10.10 (main, May 19 2023, 08:20:31) [GCC 11.3.0] Legal docs and information: aka.ms/AzureCliLegal Your CLI is up-to-date.
Additional context
Appears similar to https://github.com/Azure/azure-cli/issues/16071 but that one is closed
For az storage blob sync we are using AzCopy under the hood. Please refer here https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-authorize-azure-active-directory for the desired logged in identity. Then just call the command az storage blob sync without --auth-mode login
Hi @calvinhzy we tried what you suggested but find it does not work if the storage container has shared key access disabled. This is likely because azcopy is acquiring an account key and using that. It should still work in this scenario, however. Here is some log output describing the problem:
WARNING: There are no credentials provided in your command and environment, we will query for account key for your storage account. It is recommended to provide --connection-string, --account-key or --sas-token in your command as credentials.
and
`Cannot perform sync due to error: cannot list files due to reason -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /home/vsts/go/pkg/mod/github.com/!azure/[email protected]/azblob/zc_storage_error.go:42 ===== RESPONSE ERROR (ServiceCode=KeyBasedAuthenticationNotPermitted) ===== Description=Key based authentication is not permitted on this storage account. RequestId:b9de306f-501e-00be-6840-adbea9000000 Time:2023-07-02T23:56:23.1833127Z, Details: Code: KeyBasedAuthenticationNotPermitted GET https://REDACTED User-Agent: [AzCopy/10.13.0 Azure-Storage/0.14 (go1.16; linux)] X-Ms-Client-Request-Id: [390a4353-e273-4eae-65ae-6b4f002323d1] X-Ms-Version: [2019-12-12]
RESPONSE Status: 403 Key based authentication is not permitted on this storage account. Content-Length: [269] Content-Type: [application/xml] Date: [Sun, 02 Jul 2023 23:56:23 GMT] Server: [Microsoft-HTTPAPI/2.0] X-Ms-Error-Code: [KeyBasedAuthenticationNotPermitted] X-Ms-Request-Id: [b9de306f-501e-00be-68[40]`
please advise of any likely fix for this issue
@stephenhogg-des @dvasdekis Can you refer to the part for login in to Azure AD and make sure the role assignment is correct
https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json
https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-authorize-azure-active-directory?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json
The alternative is to call the az command with --sas-token.
@stephenhogg-des @dvasdekis Can you refer to the part for login in to Azure AD and make sure the role assignment is correct https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json
https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-authorize-azure-active-directory?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json
The alternative is to call the az command with
--sas-token.
The role assignment is correct. Also the usage of a sas token is blocked as the storage account has the above discussed setting switched on
Hi again @calvinhzy - just to repeat, the issue we are facing is not related to role assignments. It is about shared access key usage. When we allow access to a container using a shared key, there is no problem. The problem occurs when shared access key usage is disabled, because azcopy is trying to authenticate with a shared key. We should still be able to use azcopy when shared access key usage is disabled, though.
Have you tried login with azcopy with Azure AD, if shared access key is not enabled, it is hard to imagine azcopy would work without any authentication method. (sas token or account key).
Hi @calvinhzy we are able to do az storage blob upload-batch --auth-mode login without error. It is just az storage blob sync that has a problem when shared access key usage is disabled.
Hi @calvinhzy that solution does not work for us. We cannot run azcopy login because we cannot login interactively and do not have a secret or certificate. We are logging into Azure using OIDC authentication (see https://learn.microsoft.com/en-us/azure/developer/github/connect-from-azure?tabs=azure-portal%2Clinux#use-the-azure-login-action-with-openid-connect if you are unfamiliar with this), and therefore do not have a secret or certificate.
Please advise on how to fix our issue - to reiterate, it is just az storage blob sync that has a problem when shared access key usage is disabled. When shared key usage is allowed, az storage blob sync has no problems. This is because az storage blob sync is acquiring a key and using that to authenticate.
@stephenhogg-des @dvasdekis We totally understand your pain points.
Let me clarify sth here: az storage blob sync calls azcopy under the hood and pass through almost every parameter. We should support everything that azcopy supports.
azcopy supports AAD auth in two ways:
- customer sign in interactively on website
- client tool retrieves an OAuth token and then places that token into a secret store on customer's system
CLI was trying to integrate with azcopy AAD auth using the second way. But it requires different secret stores for different operating systems. It takes large efforts since CLI claims to support a lot of systems and doesn't worth doing that just for 1 or 2 commands. Besides, there was technical difficulties to retrieve all Auth info that azcopy needs (CLI is using different auth dependencies and APIs with azcopy, we can't get refresh token but azcopy requires). That's why we doesn't support --auth-mode login for az storage blob sync/az storage copy commands.
Back to your scenario, if your runner always use linux system and have a secret store (eg. Linux keyring), you could try to use the second way by yourself. I can help add AzCopy label to notify azcopy team or you can create a new issue in azure-storage-azcopy repo to ask guidance for AAD auth without interactive mode.
Sorry for all the inconvenience and hope this can help.
Hi @evelyn-ys thanks for replying. Yes, that's our understanding of the situation too. When it is possible to authenticate with a secret then az storage blob sync works nicely, as you have outlined. The issue we now have is that if shared key access is disabled for the storage account we are attempting to send files to, then sync fails. In this situation az storage blob upload-batch --auth-mode login works without problem but az storage blob sync does not - storing a key to a local Linux keyring would not fix this as key-based authentication has been deactivated anyway. Please let me know if you would like me to clarify this further.
I'm having this same issue in our CI system which runs on kubernetes and uses a user assigned managed identity with OIDC federation. Shared key access is disabled for security and only RBAC auth is allowed. To login to azure CLI we use this command, which also really shouldn't need to be done as the env vars are already in place, but that's a separate issue https://github.com/Azure/azure-cli/issues/26858
az login --federated-token "$(cat $AZURE_FEDERATED_TOKEN_FILE)" --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID
I don't see those same options in azcopy login --help. Am I missing something here?
These are the environment variables Azure AD Workload Identity injects into the kuberntes pod.
See the "Properties" drop-down at the beginning of this page:
https://azure.github.io/azure-workload-identity/docs/installation/mutating-admission-webhook.html
AZURE_AUTHORITY_HOST | The Azure Active Directory (AAD) endpoint.
AZURE_CLIENT_ID | The client ID of the AAD application or user-assigned managed identity.
AZURE_TENANT_ID | The tenant ID of the registered AAD application or user-assigned managed identity.
AZURE_FEDERATED_TOKEN_FILE | The path of the projected service account token file. /var/run/secrets/azure/tokens/azure-identity-token
Can the az or azcopy CLI tool simply use the DefaultAzureCredential Class and transparently handle auth without even needing any extra login commands or take these same vars as CLI arguments?
I've also tried setting export AZCOPY_AUTO_LOGIN_TYPE=AZCLI after the login command above, which produces this error:
$ azcopy copy testfile.txt 'https://storage-account-here.blob.core.windows.net/test-container/test.txt'
INFO: Scanning...
Failed to perform Auto-login: AzureCLICredential: WARNING: Could not retrieve credential from local cache for service principal xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx under tenant common. Trying credential under tenant xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, assuming that is an app credential.
ERROR: AADSTS700016: Application with identifier 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' was not found in the directory 'Microsoft'. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant. Trace ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Correlation ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Timestamp: 2024-01-26 12:22:01Z
Interactive authentication is needed. Please run:
az login
.
It looks like this is still an ongoing issue, my team is facing the same blocking point. Is there an update on this? Thank you!
Is there any update on this issue? We are blocked by the lack of the --auth-mode login flag and as far as I can tell, many people have complained about it since 4 years ago (when the original issue was created).
