`az network bastion tunnel` crashes with "Unexpected internal error"

[justinmchase]

Describe the bug

I am attempting to tunnel to Azure Cosmos MongoDB with bastion. I believe I have the subnet and NSG setup correctly and have added cosmos to the bastion subnet correctly, but when I connect to the tunnel the process crashes with an "Unexpected internal error" and the tunnel fails.

Command Name az network bastion tunnel

Errors:

Exception in thread Thread-1 (_start_tunnel):
Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.10.6_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/local/Cellar/[email protected]/3.10.6_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/network/custom.py", line 8601, in _start_tunnel
    tunnel_server.start_server()
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/network/tunnel.py", line 184, in start_server
    self._listen()
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/network/tunnel.py", line 117, in _listen
    auth_token = self._get_auth_token()
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/network/tunnel.py", line 104, in _get_auth_token
    raise exp
msrestazure.azure_exceptions.CloudError: Unexpected internal error

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

• Create Azure CosmosDB instance
• Create Bastion, Subnet, NSG, etc.
• Add Cosmos to the bastion subnet
• Open a tunnel to the cosmos db resource
• Get the connection string from the cosmosdb resource, change the hostname to localhost, connect using Mongo Compass (or any other desktop client).
• az network bastion tunnel --name {} --resource-group {} --target-resource-id {} --resource-port {} --port {}

COSMOS_ID=$(az cosmosdb list --resource-group $RES_NAME | jq -r '.[] | .id')
az network bastion tunnel \
  --name "$RES_NAME" \
  --resource-group "$RES_NAME" \
  --target-resource-id "$COSMOS_ID" \
  --resource-port 10255 \
  --port 10255

Expected Behavior

I expect the tunnel to succeed, or a clear error message is printed out explaining why it didn't succeed. Optionally, if CosmosDB is not a supported target then I would expect the tunnel command to check the --target-resource-id resource type and give a useful error message and not open the tunnel port at all.

Environment Summary

macOS-10.15.7-x86_64-i386-64bit, Darwin 19.6.0
Python 3.10.6
Installer: HOMEBREW

azure-cli 2.39.0

Extensions:
account 0.2.3
blueprint 0.3.1

Dependencies:
msal 1.18.0b1
azure-mgmt-resource 21.1.0b1

Additional Context

My company blocks all outbound network traffic that isn't port 443, so I need to tunnel to connect my local machine to the database for debugging and testing. I'm trying to utilize bastion for this purpose but its crashing and not giving appropriate feedback about whats going wrong.

[yonzhan]

route to CXP team

[LogicalNiko]

Similar issue with az network bastion ssh

Command group 'network bastion' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus

Exception in thread Thread-1 (_start_tunnel): Traceback (most recent call last): File "/usr/local/Cellar/[email protected]/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/local/Cellar/[email protected]/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/usr/local/Cellar/azure-cli/2.40.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/network/custom.py", line 8475, in _start_tunnel tunnel_server.start_server() File "/usr/local/Cellar/azure-cli/2.40.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/network/tunnel.py", line 184, in start_server self._listen() File "/usr/local/Cellar/azure-cli/2.40.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/network/tunnel.py", line 117, in _listen auth_token = self._get_auth_token()`

Environment:

MacOS Version 12.5.1 Install: Homebrew

[MarkDarwin]

Similar here from MacOS to a WindowsVM This works from Windows to Windows.

Error: Command group 'network bastion' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus The command failed with an unexpected error. Here is the traceback: cannot import name 'WinDLL' from 'ctypes' (/opt/homebrew/Cellar/[email protected]/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ctypes/init.py) Traceback (most recent call last): File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/knack/cli.py", line 233, in invoke cmd_result = self.invocation.execute(args) File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 663, in execute raise ex File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 726, in _run_jobs_serially results.append(self._run_job(expanded_arg, cmd_copy)) File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 697, in _run_job result = cmd_copy(params) File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 333, in call return self.handler(*args, **kwargs) File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler return op(**command_args) File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/network/custom.py", line 8352, in rdp_bastion_host from ._process_helper import launch_and_wait File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/network/_process_helper.py", line 9, in from ctypes import WinDLL, c_int, c_size_t, Structure, WinError, sizeof, pointer ImportError: cannot import name 'WinDLL' from 'ctypes' (/opt/homebrew/Cellar/[email protected]/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ctypes/init.py)

Environment: ProductName: macOS ProductVersion: 13.0 BuildVersion: 22A380

Install: Homebrew

azure-cli 2.42.0 core 2.42.0 telemetry 1.0.8

Extensions: account 0.2.3

Dependencies: msal 1.20.0 azure-mgmt-resource 21.1.0b1

[navba-MSFT]

@MarkDarwin Apologies for the late reply. You will see this issue if you try to run the command from the portal Azure cloud shell.

Please run the command from your local machine ( on PowerShell with AzCLI). Also, note that RDP command works best on Windows, for other OS’ you can do the tunnel command and then use the client of your choice.

Tagging another similar issue here. https://github.com/Azure/azure-cli/issues/23143 if that helps.

[navba-MSFT]

@MarkDarwin I wanted to do quick follow-up to check if you had a chance to look at my above comment. Please let us know if you had any updates on this. Awaiting your reply.

[navba-MSFT]

@MarkDarwin The action is currently pending on you to try the above plan. Please feel free to reopen this thread, once you have an update. We would be happy to help.

[MarkDarwin]

@navba-MSFT I am not running from the portal azure cloud shell, this was from the terminal on a mac.

az network bastion list -g <my_rg_here> does work so at least querying is ok

I'm not aware of the command just to establish a tunnel. If you can supply it, I will test tomorrow.

The command I am trying to run is:

az network bastion rdp --name mybastionname --resource-group mybastionrgname --target-resource-id /subscriptions/xxx-2fac-4469-b0c2-xxx/resourceGroups/myvmrgname/providers/Microsoft.Compute/virtualMachines/my-vm-here

If it's possible to make a connection to the bastion then use e.g. 'Microsoft Remote Desktop' application to rdp, I'd be happy with the two step process.

[elfazzz]

@navba-MSFT

I have the same issue with Linux azure-cli, 2.44.1, running on my local machine

• Create tunnel:

╰> az network bastion tunnel --name ${BASTION} --resource-group ${BRG} --target-resource-id ${RID} --resource-port 80 --port 8080
Command group 'network bastion' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Opening tunnel on port: 8080
Tunnel is ready, connect on port 8080
Ctrl + C to close

• Launch a connection test from another terminal:

╰> nc -v localhost 8080
Connection to localhost (127.0.0.1) 8080 port [tcp/http-alt] succeeded!

• Error + tunnel crash as soon as connection test is launched:

Exception in thread Thread-1 (_start_tunnel):
Traceback (most recent call last):
  File "/opt/az/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/opt/az/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/command_modules/network/custom.py", line 8404, in _start_tunnel
    tunnel_server.start_server()
  File "/opt/az/lib/python3.10/site-packages/azure/cli/command_modules/network/tunnel.py", line 184, in start_server
    self._listen()
  File "/opt/az/lib/python3.10/site-packages/azure/cli/command_modules/network/tunnel.py", line 117, in _listen
    auth_token = self._get_auth_token()
  File "/opt/az/lib/python3.10/site-packages/azure/cli/command_modules/network/tunnel.py", line 104, in _get_auth_token
    raise exp
msrestazure.azure_exceptions.CloudError: Unexpected internal error

Bastion has tunneling enabled. Target resource ID that I used here is a container instance, but I had same result using VM as destination. I got an "nc -l 80" running inside the container, port redirection is enabled in container instance, but that seems irrelevant since tunnel behaves exactly the same if I try to redirect to a closed port.

Can this issue be reopened?

[justinmchase]

@navba-MSFT Why was this closed with no fix? Also the reproduction steps you seemed to list are not really relevant to the issue.

I just want to reiterate that this issue is specifically related to tunnelling to a resource that is not a VM (CosmosDB). The error message does not seem to check or make clear the reason why tunneling to an unsupported resource type fails.

[navba-MSFT]

@justinmchase

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @aznetsuppgithub.

Issue Details

---

Describe the bug

I am attempting to tunnel to Azure Cosmos MongoDB with bastion. I believe I have the subnet and NSG setup correctly and have added cosmos to the bastion subnet correctly, but when I connect to the tunnel the process crashes with an "Unexpected internal error" and the tunnel fails.

Command Name az network bastion tunnel

Errors:

Exception in thread Thread-1 (_start_tunnel):
Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.10.6_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/local/Cellar/[email protected]/3.10.6_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/network/custom.py", line 8601, in _start_tunnel
    tunnel_server.start_server()
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/network/tunnel.py", line 184, in start_server
    self._listen()
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/network/tunnel.py", line 117, in _listen
    auth_token = self._get_auth_token()
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/network/tunnel.py", line 104, in _get_auth_token
    raise exp
msrestazure.azure_exceptions.CloudError: Unexpected internal error

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

• Create Azure CosmosDB instance
• Create Bastion, Subnet, NSG, etc.
• Add Cosmos to the bastion subnet
• Open a tunnel to the cosmos db resource
• Get the connection string from the cosmosdb resource, change the hostname to localhost, connect using Mongo Compass (or any other desktop client).
• az network bastion tunnel --name {} --resource-group {} --target-resource-id {} --resource-port {} --port {}

COSMOS_ID=$(az cosmosdb list --resource-group $RES_NAME | jq -r '.[] | .id')
az network bastion tunnel \
  --name "$RES_NAME" \
  --resource-group "$RES_NAME" \
  --target-resource-id "$COSMOS_ID" \
  --resource-port 10255 \
  --port 10255

Expected Behavior

I expect the tunnel to succeed, or a clear error message is printed out explaining why it didn't succeed. Optionally, if CosmosDB is not a supported target then I would expect the tunnel command to check the --target-resource-id resource type and give a useful error message and not open the tunnel port at all.

Environment Summary

macOS-10.15.7-x86_64-i386-64bit, Darwin 19.6.0
Python 3.10.6
Installer: HOMEBREW

azure-cli 2.39.0

Extensions:
account 0.2.3
blueprint 0.3.1

Dependencies:
msal 1.18.0b1
azure-mgmt-resource 21.1.0b1

Additional Context

My company blocks all outbound network traffic that isn't port 443, so I need to tunnel to connect my local machine to the database for debugging and testing. I'm trying to utilize bastion for this purpose but its crashing and not giving appropriate feedback about whats going wrong.

Author:	justinmchase
Assignees:	-
Labels:	Network, Service Attention, question, customer-reported, needs-team-attention, Network - Bastion, Auto-Assign
Milestone:	Backlog

[navba-MSFT]

@justinmchase Thanks for getting back and clarifying the ask. I have reopened this issue and added Service team to look into this further.

@aznetsuppgithub Could you please look into this issue ? Thanks.

[navba-MSFT]

@justinmchase Please update your bastion extension by running the below command and check if that helps ?

> az extension update -n bastion

Hope this helps.

[ghost]

Hi @justinmchase. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

[ghost]

Hi @justinmchase, since you haven’t asked that we “/unresolve” the issue, we’ll close this out. If you believe further discussion is needed, please add a comment “/unresolve” to reopen the issue.

[SolidJuho]

Hey, This issue is still /unresolve

Tested steps before, eg. install powershell, update bastion extension, but these do not resolve error.

PS /Users/juho> az network bastion rdp --name bastion-prod --resource-group prod --target-resource-id 1-prod Command group 'az network' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus The command failed with an unexpected error. Here is the traceback: cannot import name 'WinDLL' from 'ctypes' (/opt/homebrew/Cellar/[email protected]/3.10.11/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ctypes/__init__.py) Traceback (most recent call last): File "/opt/homebrew/Cellar/azure-cli/2.47.0/libexec/lib/python3.10/site-packages/knack/cli.py", line 233, in invoke cmd_result = self.invocation.execute(args) File "/opt/homebrew/Cellar/azure-cli/2.47.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute raise ex File "/opt/homebrew/Cellar/azure-cli/2.47.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially results.append(self._run_job(expanded_arg, cmd_copy)) File "/opt/homebrew/Cellar/azure-cli/2.47.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job result = cmd_copy(params) File "/opt/homebrew/Cellar/azure-cli/2.47.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__ return self.handler(*args, **kwargs) File "/opt/homebrew/Cellar/azure-cli/2.47.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler return op(**command_args) File "/Users/juho/.azure/cliextensions/bastion/azext_bastion/custom.py", line 232, in rdp_bastion_host from ._process_helper import launch_and_wait File "/Users/juho/.azure/cliextensions/bastion/azext_bastion/_process_helper.py", line 9, in <module> from ctypes import WinDLL, c_int, c_size_t, Structure, WinError, sizeof, pointer ImportError: cannot import name 'WinDLL' from 'ctypes' (/opt/homebrew/Cellar/[email protected]/3.10.11/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ctypes/__init__.py) To check existing issues, please visit: https://github.com/Azure/azure-cli/issues To open a new issue, please run az feedback``

[ghost]

Hi SolidJuho, only the original author of the issue can ask that it be unresolved. Please open a new issue with your scenario and details if you would like to discuss this topic with the team.

[deepforu47]

I am also facing same issue. Able to make tunnel connection, but as soon as trying to connect its crashing.

Opening tunnel on port: 3389 Tunnel is ready, connect on port 3389 Ctrl + C to close Exception in thread Thread-1 (_start_tunnel): Traceback (most recent call last): File "/usr/local/Cellar/[email protected]/3.10.12_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/local/Cellar/[email protected]/3.10.12_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/Users/kulsharm2/.azure/cliextensions/bastion/azext_bastion/custom.py", line 335, in _start_tunnel tunnel_server.start_server() File "/Users/kulsharm2/.azure/cliextensions/bastion/azext_bastion/tunnel.py", line 194, in start_server self._listen() File "/Users/kulsharm2/.azure/cliextensions/bastion/azext_bastion/tunnel.py", line 123, in _listen auth_token = self._get_auth_token() File "/Users/kulsharm2/.azure/cliextensions/bastion/azext_bastion/tunnel.py", line 112, in _get_auth_token self.last_token = response_json["authToken"] KeyError: 'authToken

[jiasli]

ImportError: cannot import name 'WinDLL' from 'ctypes' is tracked by #23143

[joaomariocosta]

@deepforu47 where you able to fix it? I am getting exactly the same issue in the same scenario. My colleagues are able to execute all with the same data, but in my laptop i am getting the "authToken" issue and break my tunnel connection.