az storage fs directory upload with login authentication fails with MSI aut not yet supported

[sgautrin]

This is autogenerated. Please review and update as needed.

Describe the bug

Command Name az storage fs directory upload

Errors:

The command failed with an unexpected error. Here is the traceback:
MSI auth not yet supported.
Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/azcopy/util.py", line 108, in login_auth_for_azcopy
    token_info = _unserialize_non_msi_token_payload(token_info)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/azcopy/util.py", line 144, in _unserialize_non_msi_token_payload
    parsed_authority = urlparse(token_info['_authority'])
KeyError: '_authority'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 657, in execute
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 720, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 712, in _run_job
    return cmd_copy.exception_handler(ex)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/__init__.py", line 371, in new_handler
    first(ex)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/__init__.py", line 370, in new_handler
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 691, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/operations/azcopy.py", line 74, in storage_fs_directory_copy
    azcopy = _azcopy_login_client(cmd)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/operations/azcopy.py", line 125, in _azcopy_login_client
    return AzCopy(creds=login_auth_for_azcopy(cmd))
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/azcopy/util.py", line 110, in login_auth_for_azcopy
    raise Exception('MSI auth not yet supported.')
Exception: MSI auth not yet supported.

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

• authentication with az login --use-device-code using an AD account
• az storage fs directory upload --file-system {} --account-name {} --auth-mode {} --destination-path {} --recursive --source {}

Expected Behavior

Directory to be uploaded successfully. It worked with version 2.27.2 of az cli on August 20th.

Environment Summary

Linux-4.9.0-13-amd64-x86_64-with-debian-9.13, Debian GNU/Linux 9 (stretch)
Python 3.6.10
Installer: DEB

azure-cli 2.30.0

Extensions:
storage-preview 0.7.4

Additional Context

target is an ADLS Gen 2 Data Lake

[yonzhan]

storage

[wilsonwang213]

Meet the same issue when calling "az storage blob directory upload" azure cli version : 2.30.0 storage-preview version : 0.7.1

Error: WARNING: This command is implicitly deprecated because command group 'storage blob directory' is deprecated and will be removed in a future release. Use 'az storage fs directory' instead. ERROR: The command failed with an unexpected error. Here is the traceback: ERROR: MSI auth not yet supported. Traceback (most recent call last): File "C:\Users\shiswang.azure\cliextensions\storage-preview\azext_storage_preview\azcopy\util.py", line 81, in blob_client_auth_for_azcopy token_info = _unserialize_non_msi_token_payload(token_info) File "C:\Users\shiswang.azure\cliextensions\storage-preview\azext_storage_preview\azcopy\util.py", line 104, in _unserialize_non_msi_token_payload parsed_authority = urlparse(token_info['_authority']) KeyError: '_authority'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 231, in invoke File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 657, in execute File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 720, in run_jobs_serially File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 712, in run_job File "C:\Users\shiswang.azure\cliextensions\storage-preview\azext_storage_preview_init.py", line 252, in new_handler handler(ex) File "C:\Users\shiswang.azure\cliextensions\storage-preview\azext_storage_preview_init.py", line 194, in handler raise ex File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 691, in _run_job File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 328, in call File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler File "C:\Users\shiswang.azure\cliextensions\storage-preview\azext_storage_preview\operations\azcopy.py", line 19, in storage_blob_upload azcopy = _azcopy_blob_client(cmd, client) File "C:\Users\shiswang.azure\cliextensions\storage-preview\azext_storage_preview\operations\azcopy.py", line 65, in _azcopy_blob_client return AzCopy(creds=blob_client_auth_for_azcopy(cmd, client)) File "C:\Users\shiswang.azure\cliextensions\storage-preview\azext_storage_preview\azcopy\util.py", line 83, in blob_client_auth_for_azcopy raise Exception('MSI auth not yet supported.') Exception: MSI auth not yet supported. To open an issue, please run: 'az feedback'

[evelyn-ys]

az storage fs directory upload is using azcopy which is integrated inside Azure CLI. After CLI core migrated to MSAL, MSAL won't return refresh_token any more. This causes authentication failure with azcopy when users specify --auth-mode login.

Please try these workarounds:

• az storage fs directory --file-system {} --account-name {} --destination-path {} --recursive --source {}
• az storage fs directory --file-system {} --account-name {} --destination-path {} --recursive --source {} --auth-mode key –account-key {}
• az storage fs directory --file-system {} --account-name {} --destination-path {} --recursive --source {} --connection-string {}
• az storage fs directory --file-system {} --account-name {} --destination-path {} --recursive --source {} --sas-token {}

We are working with MSAL team to see if they can add refresh_token in response and seek advice from Azcopy team if they can use MSAL cache directly.

[sgautrin]

Thank you for the suggestions, however authenticating with an AAD account is what I need, as in my context I currently don't have the other authentication options. ACLs are setup on my AAD account and I can't generate a SAS token, nor do have I access to an account key for the target storage.

In my case where I needed to transfer files from a remote server to the storage, I ended up having to download them to my desktop and upload them to the storage through Azure Storage Explorer :(

[evelyn-ys]

Sorry for the inconvenience. If you have to auth with AAD, can you try azcopy directly instead of using cli's integrated azcopy?

[sgautrin]

Sorry for the inconvenience. If you have to auth with AAD, can you try azcopy directly instead of using cli's integrated azcopy?

Ran a quick test and it did work with direct azcopy, thanks for the suggestion :)

[lubomir-angelov]

For me it worked to provide the AZCOPY vars for SP client secret login, since the fs directory uses az copy internally this resolved the auth issue for me.

PS: I am using 'az storage fs directory download' exporting these before hand works:

export AZCOPY_AUTO_LOGIN_TYPE=SPN export AZCOPY_SPA_APPLICATION_ID=<some_sp_app_id> export AZCOPY_SPA_CLIENT_SECRET=<some_password> export AZCOPY_TENANT_ID=<some_tenand_id>

[JamesBurnside]

Also getting this issue trying to run az storage azcopy after using OIDC login (https://github.com/Azure/communication-ui-library/actions/runs/9034838453/job/24828300265#step:19:70)