azure-cli
azure-cli copied to clipboard
Azure
Failed to establish a new connection (user-defined metadata / tags on blob not getting displayed)
Related command a) az storage fs file metadata show -p {path} -f {filesystem} --account-name {account-name} --auth-mode login b) az storage blob tag list -n {path} --container-name {filesystem/container_name} --account-name {account-name} --auth-mode login c) az storage blob filter --tag-filter "@container='{container_name}' and "{key}"='{value}'" --query "[].name" --output tsv --only-show-errors --account-name {account-name} --auth-mode login
Describe the bug
I want to be able to view the key and tag values using the CLI command (a) or (b) given the blob specified by {path}.
Or, list the blobs meeting the tag-filter conditions in (c).
Error message displayed: a) <urllib3.connection.HTTPSConnection object at 0x7fbdd5228190>: Failed to establish a new connection: [Errno -2] Name or service not known b) <urllib3.connection.HTTPSConnection object at 0x7f351d03d7e0>: Failed to establish a new connection: [Errno -2] Name or service not known c) <urllib3.connection.HTTPSConnection object at 0x7fb0eb30e200>: Failed to establish a new connection: [Errno -2] Name or service not known
To Reproduce Login to the VM with the customised role (see permission granted below): az login --identity Run either of the commands above (a) or (b) or (c)
Expected behavior The command should display the key and values for a) Metadata b) Blob index tags or c) list the blobs meeting the tag-filter conditions
Environment summary { "azure-cli": "2.39.0", "azure-cli-core": "2.39.0", "azure-cli-telemetry": "1.0.6", "extensions": { "storage-blob-preview": "0.6.1" } }
Additional context Permissions: Microsoft.Storage/storageAccounts/fileServices/read Microsoft.Storage/storageAccounts/blobServices/containers/read Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read Microsoft.Storage/storageAccounts/blobServices/containers/blobs/tags/read Microsoft.Storage/storageAccounts/blobServices/containers/blobs/filter/action
az storage blob tag list command works for common storage account not ADLS Gen2 account.
But az storage fs file metadata show should work for ADLS Gen2 account:
Can you confirm that the filesystem and path exist?
Thank you for your feedback. This has been routed to the support team for assistance.
@evelyn-ys,
Thank you for your reply.
Storage account is not ADLS Gen2.
Yes, the path exist as I can use other commands (e.g. az storage blob list).
Regards,
@montanoeuse Thanks for confirming that you are using non-ADLS gen 2 storage account. I tried to create a repro using Standard NonHNS storage account and I was able to get the required information:
I am sharing the command here if that helps:
az storage blob metadata show --container-name MyContainerName --name MyBlobName.txt { "test": "myvalue" }
az storage blob tag list -n new.txt --container-name MyContainerName --account-name MyAccountName { "IsLatest": "true", "test": "one" }
az storage blob filter --account-name MyAccountName --tag-filter ""@container"='MyContainerName' and "IsLatest"='true' and "test"='one'" [ { "containerName": "MyContainerName", "name": "MyBlobName.txt", "tags": { "IsLatest": "true", "test": "one" } } ]
Are there any proxy / firewall in your environment ? If yes, could you please try it from a different network ?
Could you please run the same above command with the --debug switch and share the complete output here ?
Thanks you @navba-MSFT for your reply.
And, firewall/proxy should not be causing the issue as I can run another command successfully. Below is an example.
az storage blob list --container-name {container_name} --query "[?properties.blobTier=='Cool'].name" --prefix {prefix_string} --output tsv --only-show-errors --account-name {account_name} --marker "" --auth-mode login
I had no problem with the commands you tested on another environment (not using system-assigned managed identity).
Below is the output of command (b) with debug option added:
cli.knack.cli: Command arguments: ['storage', 'blob', 'tag', 'list', '-n', ‘{path}’, '--container-name', '{container_name} ', '--account-name', '{account_name} ', '--auth-mode', 'login', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7fd1abaf7c70>, <function OutputProducer.on
_global_arguments at 0x7fd1ab850550>, <function CLIQuery.on_global_arguments at 0x7fd1ab881750>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'storage': ['azure.cli.command_modules.storage', 'azext_storage_blob_preview']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name Load Time Groups Commands
cli.azure.cli.core: storage 0.054 56 261
cli.azure.cli.core: Total (1) 0.054 56 261
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name Load Time Groups Commands Directory
cli.azure.cli.core: storage-blob-preview 0.013 10 35 /home/username/.azure/cliextensions/storage-blob-preview
cli.azure.cli.core: Total (1) 0.013 10 35
cli.azure.cli.core: Loaded 57 groups, 264 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command : storage blob tag list
cli.azure.cli.core: Command table: storage blob tag list
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7fd1aaf73490>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/username/.azure/commands/2022-09-09.12-54-41.storage_blob_tag_li
st.2857757.log'.
az_command_data_logger: command args: storage blob tag list -n {} --container-name {} --account-name {} --auth-mode {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7fd1
aad8cee0>]
cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 619, in _get_attr
op = import_module(full_mod_path)
File "/opt/az/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'azext_storage_blob_preview.vendored_sdks.azure_storage_blob.v2020_10_02.blob'
cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 619, in _get_attr
op = import_module(full_mod_path)
File "/opt/az/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'azext_storage_blob_preview.vendored_sdks.azure_storage_blob.v2020_10_02.file'
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7fd1aae31360>, <functi
on register_cache_arguments.<locals>.add_cache_arguments at 0x7fd1aae31480>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7fd1ab8505e0>, <function CLIQuery.handle_q
uery_parameter at 0x7fd1ab8817e0>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7fd1aae313f0>]
az_command_data_logger: extension name: storage-blob-preview
az_command_data_logger: extension version: 0.6.1
Command group 'storage blob tag' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
urllib3.connectionpool: Starting new HTTP connection (1): 169.254.169.254:80
urllib3.connectionpool: http://169.254.169.254:80 "GET /metadata/identity/oauth2/token?resource=https%3A%2F%2Fstorage.azure.com&api-version=2018-02-0
1 HTTP/1.1" 200 1671
msrestazure.azure_active_directory: MSI: Retrieving a token from http://169.254.169.254/metadata/identity/oauth2/token, with payload {'resource': 'ht
tps://storage.azure.com', 'api-version': '2018-02-01'}
msrestazure.azure_active_directory: MSI: Token retrieved
cli.azure.cli.core.auth.adal_authentication: MSIAuthenticationWrapper.get_token: scopes=('https://storage.azure.com/.default',), kwargs={}
msrestazure.azure_active_directory: MSI: token is found in cache.
cli.azure.cli.core.auth.adal_authentication: Normalize expires_on: '1662771150' -> 1662771150
urllib3.connectionpool: Starting new HTTPS connection (1): {account_name} .blob.core.windows.net:443
urllib3.connectionpool: Starting new HTTPS connection (2): {account_name} .blob.core.windows.net:443
urllib3.connectionpool: Starting new HTTPS connection (3): {account_name} .blob.core.windows.net:443
urllib3.connectionpool: Starting new HTTPS connection (4): {account_name} .blob.core.windows.net:443
cli.azure.cli.core.util: azure.cli.core.util.handle_exception is called with an exception:
cli.azure.cli.core.util: Traceback (most recent call last):
File "/opt/az/lib/python3.10/site-packages/knack/cli.py", line 231, in invoke
cmd_result = self.invocation.execute(args)
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
raise ex
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 718, in _run_job
return cmd_copy.exception_handler(ex)
File "/home/username/.azure/cliextensions/storage-blob-preview/azext_storage_blob_preview/__init__.py", line 265, in new_handler
raise ex
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
result = cmd_copy(params)
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
return self.handler(*args, **kwargs)
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
return op(**command_args)
File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 73, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/username/.azure/cliextensions/storage-blob-preview/azext_storage_blob_preview/vendored_sdks/azure_storage_blob/v2020_10_02/_blob_client
.py", line 2811, in get_blob_tags
_, tags = self._client.blob.get_tags(**options)
File "/home/username/.azure/cliextensions/storage-blob-preview/azext_storage_blob_preview/vendored_sdks/azure_storage_blob/v2020_10_02/_generated/o
perations/_blob_operations.py", line 3321, in get_tags
pipeline_response = self._client._pipeline.run(request, stream=False, **kwargs)
File "/opt/az/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 211, in run
return first_node.send(pipeline_request) # type: ignore
File "/opt/az/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 71, in send
response = self.next.send(request)
File "/opt/az/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 71, in send
response = self.next.send(request)
File "/opt/az/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 71, in send
response = self.next.send(request)
[Previous line repeated 2 more times]
File "/opt/az/lib/python3.10/site-packages/azure/core/pipeline/policies/_redirect.py", line 158, in send
response = self.next.send(request)
File "/opt/az/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 71, in send
response = self.next.send(request)
File "/home/username/.azure/cliextensions/storage-blob-preview/azext_storage_blob_preview/vendored_sdks/azure_storage_blob/v2020_10_02/_shared/poli
cies.py", line 515, in send
raise err
File "/home/username/.azure/cliextensions/storage-blob-preview/azext_storage_blob_preview/vendored_sdks/azure_storage_blob/v2020_10_02/_shared/poli
cies.py", line 489, in send
response = self.next.send(request)
File "/opt/az/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 71, in send
response = self.next.send(request)
File "/opt/az/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 71, in send
response = self.next.send(request)
File "/opt/az/lib/python3.10/site-packages/azure/core/pipeline/policies/_authentication.py", line 119, in send
response = self.next.send(request)
File "/opt/az/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 71, in send
response = self.next.send(request)
File "/home/username/.azure/cliextensions/storage-blob-preview/azext_storage_blob_preview/vendored_sdks/azure_storage_blob/v2020_10_02/_shared/poli
cies.py", line 290, in send
response = self.next.send(request)
File "/opt/az/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 71, in send
response = self.next.send(request)
File "/opt/az/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 71, in send
response = self.next.send(request)
File "/opt/az/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 103, in send
self._sender.send(request.http_request, **request.context.options),
File "/home/username/.azure/cliextensions/storage-blob-preview/azext_storage_blob_preview/vendored_sdks/azure_storage_blob/v2020_10_02/_shared/base
_client.py", line 333, in send
return self._transport.send(request, **kwargs)
File "/opt/az/lib/python3.10/site-packages/azure/core/pipeline/transport/_requests_basic.py", line 361, in send
raise error
azure.core.exceptions.ServiceRequestError: <urllib3.connection.HTTPSConnection object at 0x7fd1a93dd780>: Failed to establish a new connection: [Errn
o -2] Name or service not known
cli.azure.cli.core.azclierror: <urllib3.connection.HTTPSConnection object at 0x7fd1a93dd780>: Failed to establish a new connection: [Errno -2] Name o
r service not known
az_command_data_logger: <urllib3.connection.HTTPSConnection object at 0x7fd1a93dd780>: Failed to establish a new connection: [Errno -2] Name or servi
ce not known
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7fd1aaf736d0>]
az_command_data_logger: exit code: 1
cli.__main__: Command ran in 84.018 seconds (init: 0.132, invoke: 83.885)
telemetry.save: Save telemetry record of length 3309 in cache
telemetry.check: Negative: The /home/username/.azure/telemetry.txt was modified at 2022-09-09 12:51:57.325144, which in less than 600.000000 s
@montanoeuse The error seems to be specific to establishing the network connection. Also since you confirmed that the same commands work fine on another environment, It seems like the issue is only related to your enviornment. I could see the below in your debug output:
urllib3.connectionpool: Starting new HTTPS connection (1): {account_name} .blob.core.windows.net:443
Could you confirm if you had entered the right account name ?
Could you please send me the complete debug output over email as-is (without modifying the parameters) ? Here is my email: navba @ Microsoft . com
Awaiting your reply.
Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.
Issue Details
Related command a) az storage fs file metadata show -p {path} -f {filesystem} --account-name {account-name} --auth-mode login b) az storage blob tag list -n {path} --container-name {filesystem/container_name} --account-name {account-name} --auth-mode login c) az storage blob filter --tag-filter "@container='{container_name}' and "{key}"='{value}'" --query "[].name" --output tsv --only-show-errors --account-name {account-name} --auth-mode login
Describe the bug
I want to be able to view the key and tag values using the CLI command (a) or (b) given the blob specified by {path}.
Or, list the blobs meeting the tag-filter conditions in (c).
Error message displayed: a) <urllib3.connection.HTTPSConnection object at 0x7fbdd5228190>: Failed to establish a new connection: [Errno -2] Name or service not known b) <urllib3.connection.HTTPSConnection object at 0x7f351d03d7e0>: Failed to establish a new connection: [Errno -2] Name or service not known c) <urllib3.connection.HTTPSConnection object at 0x7fb0eb30e200>: Failed to establish a new connection: [Errno -2] Name or service not known
To Reproduce Login to the VM with the customised role (see permission granted below): az login --identity Run either of the commands above (a) or (b) or (c)
Expected behavior The command should display the key and values for a) Metadata b) Blob index tags or c) list the blobs meeting the tag-filter conditions
Environment summary { "azure-cli": "2.39.0", "azure-cli-core": "2.39.0", "azure-cli-telemetry": "1.0.6", "extensions": { "storage-blob-preview": "0.6.1" } }
Additional context Permissions: Microsoft.Storage/storageAccounts/fileServices/read Microsoft.Storage/storageAccounts/blobServices/containers/read Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read Microsoft.Storage/storageAccounts/blobServices/containers/blobs/tags/read Microsoft.Storage/storageAccounts/blobServices/containers/blobs/filter/action
| Author: | montanoeuse |
|---|---|
| Assignees: | - |
| Labels: |
|
| Milestone: | Backlog |
@montanoeuse Thanks for sharing all the debug logs details over email. I am adding the Service team to assist on this further.
We have further isolated that --auth-mode key and with AZURE_STORAGE_KEY and AZURE_STORAGE_ACCOUNT environment variables it works fine. We have also checked that the port is open and connectivity is successful. Also the nslookup is working fine.
This successful test was performed on the same environment where it fails while listing the tag.
@xgithubtriage Could you please look into tis further and provide an update on this ? Awaiting your reply.