azure-cli
azure-cli copied to clipboard
Published
20 hours ago •
Azure
Azure
`az network bastion tunnel` quietly drops connection
Describe the bug
Having tunneled to an Azure instance, the az network bastion tunnel command drops the connection. Rather than raising an exception or reconnecting, it hangs. It is only possible to identify that there was a problem when debug mode is enabled. Without debug mode, the connection is dropped silently. The interval varies between a few minutes to several hours.
Related command
az network bastion tunnel \
--debug \
--name <redacted> \
--resource-group <redacted> \
--resource-port 22 \
--port 2222 \
--target-resource-id "/subscriptions/<redacted>/resourceGroups/<redacted>/providers/Microsoft.Compute/virtualMachines/<redacted>" \
2> az-network-bastion-tunnel.log
Contents of az-network-bastion-tunnel.log are shown below.
Errors
No errors raised.
Issue script & Debug output
Logs
DEBUG: cli.knack.cli: Command arguments: ['network', 'bastion', 'tunnel', '--debug', '--name', '<name>', '--resource-group', '<resource group>', '--resource-port', '22', '--port', '2222', '--target-resource-id', '/subscriptions/<subscription id>/resourceGroups/<resource group>/providers/Microsoft.Compute/virtualMachines/<machine name>']
DEBUG: cli.knack.cli: __init__ debug log:
Cannot enable color.
DEBUG: cli.knack.cli: Event: Cli.PreExecute []
DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x<redacted>>, <function OutputProducer.on_global_arguments at 0x<redacted>>, <function CLIQuery.on_global_arguments at 0x<redacted>>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
DEBUG: cli.azure.cli.core: Modules found from index for 'network': ['azure.cli.command_modules.network', 'azure.cli.command_modules.privatedns', 'azext_bastion']
DEBUG: cli.azure.cli.core: Loading command modules:
DEBUG: cli.azure.cli.core: Name Load Time Groups Commands
DEBUG: cli.azure.cli.core: network 0.349 115 355
DEBUG: cli.azure.cli.core: privatedns 0.009 14 60
DEBUG: cli.azure.cli.core: Total (2) 0.358 129 415
DEBUG: cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
DEBUG: cli.azure.cli.core: Loading extensions:
DEBUG: cli.azure.cli.core: Name Load Time Groups Commands Directory
DEBUG: cli.azure.cli.core: bastion 0.004 2 9 <redacted>/cliextensions/bastion
DEBUG: cli.azure.cli.core: Total (1) 0.004 2 9
DEBUG: cli.azure.cli.core: Loaded 129 groups, 424 commands.
DEBUG: cli.azure.cli.core: Found a match in the command table.
DEBUG: cli.azure.cli.core: Raw command : network bastion tunnel
DEBUG: cli.azure.cli.core: Command table: network bastion tunnel
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x<redacted>>]
DEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '<redacted>/commands/2024-02-13.12-43-30.network_bastion_tunnel.51973.log'.
INFO: az_command_data_logger: command args: network bastion tunnel --debug --name {} --resource-group {} --resource-port {} --port {} --target-resource-id {}
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x<redacted>>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x<redacted>>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x<redacted>>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x<redacted>>, <function CLIQuery.handle_query_parameter at 0x<redacted>>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x<redacted>>]
INFO: az_command_data_logger: extension name: bastion
INFO: az_command_data_logger: extension version: 0.2.3
WARNING: Command group 'az network' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
DEBUG: cli.azure.cli.core.auth.persistence: build_persistence: location='<redacted>/msal_token_cache.json', encrypt=False
DEBUG: cli.azure.cli.core.auth.binary_cache: load: <redacted>/msal_http_cache.bin
DEBUG: urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
DEBUG: msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/<redacted>/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/<redacted>/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/<redacted>/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/<redacted>/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/<redacted>/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/<redacted>/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/<redacted>/kerberos', 'tenant_region_scope': 'NA', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
DEBUG: msal.application: Broker enabled? False
DEBUG: cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={}
DEBUG: cli.azure.cli.core.auth.msal_authentication: UserCredential.get_token: scopes=('https://management.core.windows.net//.default',), claims=None, kwargs={}
DEBUG: msal.application: Cache hit an AT
DEBUG: msal.telemetry: Generate or reuse correlation_id: <redacted>
DEBUG: cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/<subscription id>/resourceGroups/<resource group>/providers/Microsoft.Network/bastionHosts/<name>?api-version=2022-01-01'
DEBUG: cli.azure.cli.core.sdk.policies: Request method: 'GET'
DEBUG: cli.azure.cli.core.sdk.policies: Request headers:
DEBUG: cli.azure.cli.core.sdk.policies: 'Accept': 'application/json'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': '<redacted>'
DEBUG: cli.azure.cli.core.sdk.policies: 'CommandName': 'network bastion tunnel'
DEBUG: cli.azure.cli.core.sdk.policies: 'ParameterSetName': '--debug --name --resource-group --resource-port --port --target-resource-id'
DEBUG: cli.azure.cli.core.sdk.policies: 'User-Agent': 'AZURECLI/2.57.0 (HOMEBREW) (AAZ) azsdk-python-core/1.28.0 Python/3.11.7 (macOS-14.3-arm64-arm-64bit)'
DEBUG: cli.azure.cli.core.sdk.policies: 'Authorization': '*****'
DEBUG: cli.azure.cli.core.sdk.policies: Request body:
DEBUG: cli.azure.cli.core.sdk.policies: This request has no body
DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
DEBUG: urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/<subscription id>/resourceGroups/<resource group>/providers/Microsoft.Network/bastionHosts/<name>?api-version=2022-01-01 HTTP/1.1" 200 1701
DEBUG: cli.azure.cli.core.sdk.policies: Response status: 200
DEBUG: cli.azure.cli.core.sdk.policies: Response headers:
DEBUG: cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache'
DEBUG: cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache'
DEBUG: cli.azure.cli.core.sdk.policies: 'Content-Length': '1701'
DEBUG: cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json; charset=utf-8'
DEBUG: cli.azure.cli.core.sdk.policies: 'Expires': '-1'
DEBUG: cli.azure.cli.core.sdk.policies: 'ETag': '<redacted>'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '<redacted>'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': '<redacted>'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-arm-service-request-id': '<redacted>'
DEBUG: cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-reads': '11999'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'EASTUS2:20240213T174330Z:<redacted>'
DEBUG: cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff'
DEBUG: cli.azure.cli.core.sdk.policies: 'X-Cache': 'CONFIG_NOCACHE'
DEBUG: cli.azure.cli.core.sdk.policies: 'X-MSEdge-Ref': 'Ref A: <redacted> Ref B: <redacted> Ref C: 2024-02-13T17:43:30Z'
DEBUG: cli.azure.cli.core.sdk.policies: 'Date': 'Tue, 13 Feb 2024 17:43:30 GMT'
DEBUG: cli.azure.cli.core.sdk.policies: Response content:
DEBUG: cli.azure.cli.core.sdk.policies: {
"name": "<name>",
"id": "/subscriptions/<subscription id>/resourceGroups/<resource group>/providers/Microsoft.Network/bastionHosts/<name>",
"etag": "W/\"<redacted>\"",
"type": "Microsoft.Network/bastionHosts",
"location": "eastus2",
"tags": {
"User": "<redacted>"
},
"properties": {
"provisioningState": "Succeeded",
"dnsName": "bst-<redacted>.bastion.azure.com",
"scaleUnits": 2,
"enableTunneling": true,
"enableIpConnect": false,
"disableCopyPaste": false,
"enableShareableLink": false,
"enableKerberos": false,
"ipConfigurations": [
{
"name": "IpConf",
"id": "/subscriptions/<subscription id>/resourceGroups/<resource group>/providers/Microsoft.Network/bastionHosts/<name>/bastionHostIpConfigurations/IpConf",
"etag": "<redacted>",
"type": "Microsoft.Network/bastionHosts/bastionHostIpConfigurations",
"properties": {
"provisioningState": "Succeeded",
"privateIPAllocationMethod": "Dynamic",
"publicIPAddress": {
"id": "/subscriptions/<subscription id>/resourceGroups/<resource group>/providers/Microsoft.Network/publicIPAddresses/<redacted>"
},
"subnet": {
"id": "/subscriptions/<subscription id>/resourceGroups/<resource group>/providers/Microsoft.Network/virtualNetworks/<redacted>/subnets/AzureBastionSubnet"
}
}
}
]
},
"sku": {
"name": "Standard"
}
}
INFO: cli.azext_bastion.tunnel: Port 2222 is open
INFO: cli.azext_bastion.tunnel: Creating a socket on port: 2222
INFO: cli.azext_bastion.tunnel: Setting socket options
INFO: cli.azext_bastion.tunnel: Binding to socket on local address and port
INFO: cli.azext_bastion.tunnel: Finished initialization
WARNING: cli.azext_bastion.custom: Opening tunnel on port: 2222
WARNING: cli.azext_bastion.custom: Tunnel is ready, connect on port 2222
WARNING: cli.azext_bastion.custom: Ctrl + C to close
DEBUG: urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
DEBUG: msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/<redacted>/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/<redacted>/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/<redacted>/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/<redacted>/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/<redacted>/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/<redacted>/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/<redacted>/kerberos', 'tenant_region_scope': 'NA', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
DEBUG: msal.application: Broker enabled? False
DEBUG: cli.azure.cli.core.auth.msal_authentication: UserCredential.get_token: scopes=('https://management.core.windows.net//.default',), claims=None, kwargs={}
DEBUG: msal.application: Cache hit an AT
DEBUG: msal.telemetry: Generate or reuse correlation_id: <redacted>
DEBUG: cli.azext_bastion.tunnel: Content: {'resourceId': '/subscriptions/<subscription id>/resourceGroups/<resource group>/providers/Microsoft.Compute/virtualMachines/<machine name>', 'protocol': 'tcptunnel', 'workloadHostPort': '22', 'aztoken': '<redacted>', 'token': None}
DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): bst-<redacted>.bastion.azure.com:443
DEBUG: urllib3.connectionpool: https://bst-<redacted>.bastion.azure.com:443 "POST /api/tokens HTTP/1.1" 200 None
INFO: cli.azext_bastion.tunnel: Websocket, connected status: True
INFO: cli.azext_bastion.tunnel: Got debugger connection... index: 1
INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1
INFO: cli.azext_bastion.tunnel: Both debugger and websocket threads started...
INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 21, index: 1
INFO: cli.azext_bastion.tunnel: Successfully connected to local server..
INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1
INFO: cli.azext_bastion.tunnel: Received websocket index: 1
INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1
INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 1504, index: 1
INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1
INFO: cli.azext_bastion.tunnel: Received websocket index: 1
INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1
INFO: cli.azext_bastion.tunnel: Received websocket index: 1
INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1
INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 48, index: 1
INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1
INFO: cli.azext_bastion.tunnel: Received websocket index: 1
INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1
INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 16, index: 1
INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1
INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 44, index: 1
INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1
INFO: cli.azext_bastion.tunnel: Received websocket index: 1
INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1
INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 68, index: 1
INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1
INFO: cli.azext_bastion.tunnel: Received websocket index: 1
INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1
INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 140, index: 1
INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1
INFO: cli.azext_bastion.tunnel: Received websocket index: 1
<100k lines redacted all having the same pattern>
INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1
INFO: cli.azext_bastion.tunnel: Received websocket index: 1
INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1
INFO: cli.azext_bastion.tunnel: Received websocket index: 1
INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1
INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 80, index: 1
INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1
INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 84, index: 1
INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1
INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 84, index: 1
INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1
INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 84, index: 1
INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1
INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 84, index: 1
INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1
INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 84, index: 1
INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1
INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 84, index: 1
INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1
INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 100, index: 1
INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1
INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 84, index: 1
INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1
INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 84, index: 1
INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1
INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1
INFO: cli.azext_bastion.tunnel: Connection to remote host was lost.
INFO: cli.azext_bastion.tunnel: Client disconnected!, index: 1
INFO: cli.azext_bastion.tunnel: [Errno 9] Bad file descriptor
INFO: cli.azext_bastion.tunnel: Client disconnected 1
INFO: cli.azext_bastion.tunnel: Cleaning up session
DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): bst-<redacted>.bastion.azure.com:443
DEBUG: urllib3.connectionpool: https://bst-<redacted>.bastion.azure.com:443 "DELETE /api/tokens/<redacted> HTTP/1.1" 204 0
INFO: cli.azext_bastion.tunnel: Both debugger and websocket threads stopped...
INFO: cli.azext_bastion.tunnel: Stopped local server..
[hangs here]
Expected behavior
The connection remains stable, the connection is re-established, or an error is raised if reconnection attempts fail.
Environment Summary
azure-cli 2.57.0
core 2.57.0
telemetry 1.1.0
Extensions:
bastion 0.2.3
ssh 1.1.3
Dependencies:
msal 1.26.0
azure-mgmt-resource 23.1.0b2
Python location '/opt/homebrew/Cellar/azure-cli/2.57.0/libexec/bin/python'
Extensions directory '/Users/till/.azure/cliextensions'
Python (Darwin) 3.11.7 (main, Dec 4 2023, 18:10:11) [Clang 15.0.0 (clang-1500.1.0.2.5)]
Legal docs and information: aka.ms/AzureCliLegal
Your CLI is up-to-date.
Additional context
Running on M1 MacBook Pro (2020), macOS 14.3 (23D56).
Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @aznetsuppgithub.
![microsoft-github-policy-service[bot] avatar](https://avatars.githubusercontent.com/in/95686?v=4 "Published by a pub.dev verified publisher"){kind=small}
Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @bastionsuppgithub.
Hello,
I have almost the same issue, also on Silicon Mac: I'm opening a tunnel to a VM, and uses the tunnel to ssh to the VM.
at random internal, the az bastion tunnel drops, the ssh connection dies. The main difference is that I have an additional error line in the logs:
cli.azext_bastion.tunnel: Waiting for debugger data, index: 0
cli.azext_bastion.tunnel: [SSL] record layer failure (_ssl.c:2580)
cli.azext_bastion.tunnel: Client disconnected!, index: 0
cli.azext_bastion.tunnel: [Errno 9] Bad file descriptor
cli.azext_bastion.tunnel: Client disconnected 0
cli.azext_bastion.tunnel: Cleaning up session
urllib3.connectionpool: Starting new HTTPS connection (1): bst-92d64564-079c-489b-bce5-4522582a7c1e.bastion.azure.com:443
urllib3.connectionpool: https://bst-92d64564-079c-489b-bce5-4522582a7c1e.bastion.azure.com:443 "DELETE /api/tokens/D5EAD29EF05D3B3FA86B622CC39968D448D6AE86DD8D31E41B999E009A787DAB HTTP/1.1" 204 0
cli.azext_bastion.tunnel: Both debugger and websocket threads stopped...
cli.azext_bastion.tunnel: Stopped local server.
error: cli.azext_bastion.tunnel: [SSL] record layer failure (_ssl.c:2580)
Strangely enough:
- I have several colleagues also on Mac, using the same bastions, and they don't seem to have the issue.
- a few weeks back, before I upgraded the modules ssh and tunnel, I didn't have the issue (bastion 0.3.0 and ssh 2.0.3)
I don't think current issue is related to #24600 because I'm not creating several connections through the tunnel (can't comment if it's the case for @tillahoffmann )
Note that az bastion ssh is dying the same way at random interval
❯ az --version
azure-cli 2.65.0
core 2.65.0
telemetry 1.1.0
Extensions:
account 0.2.5
azure-devops 0.26.0
bastion 1.3.0
ssh 2.0.5
Dependencies:
msal 1.31.0
azure-mgmt-resource 23.1.1
Python location '/opt/homebrew/Cellar/azure-cli/2.65.0/libexec/bin/python'
Extensions directory '/Users/[email protected]/.azure/cliextensions'
Python (Darwin) 3.11.10 (main, Sep 7 2024, 01:03:31) [Clang 15.0.0 (clang-1500.3.9.4)]
Legal docs and information: aka.ms/AzureCliLegal
Your CLI is up-to-date.
MacBook Pro M2 Max, MacOS Sequoia 15.0.1 Connecting using VPN (Cisco secure client version 5.16.103)
I am getting this same issue, from an AWS runner. We are trying to open a tunnel, to connect to a backend VM with SSH.
We are using the command:
az network bastion tunnel \
--name "$BASTION_NAME" \
--resource-group "$AZURE_RESOURCE_GROUP" \
--target-resource-id "$VM_RESOURCE_ID" \
--port "$SSH_PORT" \
--resource-port 22 --debug > bastion_tunnel.log 2>&1 &
The tunnel opens momentarily, but then we are getting an error:
INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 0
INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 0
INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 0
INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 148, index: 0
INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 0
INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 0
INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 0
INFO: cli.azext_bastion.tunnel: Received websocket index: 0
INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 0
INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 0
INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 0
INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 0, index: 0
INFO: cli.azext_bastion.tunnel: Client close, index: 0
INFO: cli.azext_bastion.tunnel: Client disconnected 0
INFO: cli.azext_bastion.tunnel: Received websocket index: 0
INFO: cli.azext_bastion.tunnel: Websocket close, index: 0
INFO: cli.azext_bastion.tunnel: Client disconnected!, index: 0
INFO: cli.azext_bastion.tunnel: Cleaning up session
DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): REDACTED_BASTIONHOSTNAME:443
DEBUG: urllib3.connectionpool: _REDACTED_ HTTP/1.1" 204 0
INFO: cli.azext_bastion.tunnel: Both debugger and websocket threads stopped...
INFO: cli.azext_bastion.tunnel: Stopped local server..
Any updates on this issue please, @yonzhan ?
