az bastion ssh fails when missing read permissions on vm nic without throwing meaningful error

[cveld]

Describe the bug

Whenever I try to access a virtual machine through az bastion ssh, it fails when I don't have read access to the nic without throwing a meaningful error:

Exception in thread Thread-1 (_start_tunnel):
Traceback (most recent call last):
  File "threading.py", line 1016, in _bootstrap_inner
  File "threading.py", line 953, in run
  File "C:\Users\CarlintVeld\.azadmin\cliextensions\bastion\azext_bastion\custom.py", line 335, in _start_tunnel     
    tunnel_server.start_server()
  File "C:\Users\CarlintVeld\.azadmin\cliextensions\bastion\azext_bastion\tunnel.py", line 194, in start_server      
    self._listen()
  File "C:\Users\CarlintVeld\.azadmin\cliextensions\bastion\azext_bastion\tunnel.py", line 123, in _listen
    auth_token = self._get_auth_token()
  File "C:\Users\CarlintVeld\.azadmin\cliextensions\bastion\azext_bastion\tunnel.py", line 112, in _get_auth_token   
    self.last_token = response_json["authToken"]
KeyError: 'authToken'

Related command

az bastion ssh

Errors

See above.

Issue script & Debug output

N/A

Expected behavior

It should passthrough the error that is given back from the rest api, e.g.

(AuthorizationFailed) The client 'admin@suppressed' with object id 'suppressed' does not have authorization to perform action 'Microsoft.Network/networkInterfaces/read' over scope '/subscriptions/suppressed/resourceGroups/rg-myresourcegroup/providers/Microsoft.Network/networkInterfaces/nic-mynic' or the scope is invalid. If access was recently granted, please refresh your credentials.
Code: AuthorizationFailed

Environment Summary

{
  "azure-cli": "2.52.0",
  "azure-cli-core": "2.52.0",
  "azure-cli-telemetry": "1.1.0",
  "extensions": {
    "bastion": "0.2.5",
    "resource-graph": "2.1.0",
    "ssh": "2.0.1"
  }
}

Additional context

No response

[azure-client-tools-bot-prd[bot]]

Hi @cveld,

2.52.0 is not the latest Azure CLI(2.53.0).

If you haven't already attempted to do so, please upgrade to the latest Azure CLI version by following https://learn.microsoft.com/en-us/cli/azure/update-azure-cli.

[yonzhan]

Thank you for opening this issue, we will look into it.

[ThenukaD]

Facing same issue with following command connecting to AZ bastion service

thenukakeerthibandara@Thenukas-MacBook-Air ~ % az network bastion ssh
--name xxxxxxxx
--resource-group xxxxxxx
--target-resource-id "/subscriptions/xxxxxxxxxx/resourceGroups/xxxxxxxxxxxxxxxx/providers/Microsoft.Compute/virtualMachines/xxxx"
--auth-type "AAD"
--subscription xxxxxxxxxxxxx

Command group 'az network' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus /var/folders/r8/xqpjp4xs4wxb9ms_bl_v8shr0000gn/T/aadsshcertalirbsym contains sensitive information (id_rsa, id_rsa.pub). Please delete once this certificate is no longer being used. Generated SSH certificate /var/folders/r8/xqpjp4xs4wxb9ms_bl_v8shr0000gn/T/aadsshcertalirbsym/id_rsa.pub-aadcert.pub is valid until 2023-10-18 17:01:27 in local time. Exception in thread Thread-1 (_start_tunnel): Traceback (most recent call last): File "/opt/homebrew/Cellar/[email protected]/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/opt/homebrew/Cellar/[email protected]/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/Users/thenukakeerthibandara/.azure/cliextensions/bastion/azext_bastion/custom.py", line 335, in _start_tunnel tunnel_server.start_server() File "/Users/thenukakeerthibandara/.azure/cliextensions/bastion/azext_bastion/tunnel.py", line 194, in start_server self._listen() File "/Users/thenukakeerthibandara/.azure/cliextensions/bastion/azext_bastion/tunnel.py", line 123, in _listen auth_token = self._get_auth_token() File "/Users/thenukakeerthibandara/.azure/cliextensions/bastion/azext_bastion/tunnel.py", line 112, in _get_auth_token self.last_token = response_json["authToken"] KeyError: 'authToken' ^C%

Please note that version of AZ is the latest as suggested above

{ "azure-cli": "2.53.0", "azure-cli-core": "2.53.0", "azure-cli-telemetry": "1.1.0", "extensions": { "bastion": "0.2.5", "ssh": "1.1.2" }

[GoodTown]

having the same issue connecting to our VM's using the latest azure CLI version 2.53.0

Exception in thread Thread-1 (_start_tunnel): Traceback (most recent call last): File "threading.py", line 1016, in _bootstrap_inner File "threading.py", line 953, in run File "C:\Users\philippe.bonneville\.azure\cliextensions\bastion\azext_bastion\custom.py", line 335, in _start_tunnel tunnel_server.start_server() File "C:\Users\philippe.bonneville\.azure\cliextensions\bastion\azext_bastion\tunnel.py", line 194, in start_server self._listen() File "C:\Users\philippe.bonneville\.azure\cliextensions\bastion\azext_bastion\tunnel.py", line 123, in _listen auth_token = self._get_auth_token() File "C:\Users\philippe.bonneville\.azure\cliextensions\bastion\azext_bastion\tunnel.py", line 112, in _get_auth_token self.last_token = response_json["authToken"] KeyError: 'authToken'

[jaywithersbrewin]

Getting the same error myself for az network bastion ssh --name "xxxxx" --resource-group "xxxxx" --target-resource-id "xxxxx" --auth-type AAD:

Exception in thread Thread-1 (_start_tunnel): Traceback (most recent call last): File "threading.py", line 1016, in _bootstrap_inner File "threading.py", line 953, in run File "C:\Users\withers_j.azure\cliextensions\bastion\azext_bastion\custom.py", line 335, in _start_tunnel tunnel_server.start_server() File "C:\Users\withers_j.azure\cliextensions\bastion\azext_bastion\tunnel.py", line 194, in start_server self._listen() File "C:\Users\withers_j.azure\cliextensions\bastion\azext_bastion\tunnel.py", line 123, in _listen auth_token = self._get_auth_token() File "C:\Users\withers_j.azure\cliextensions\bastion\azext_bastion\tunnel.py", line 112, in _get_auth_token self.last_token = response_json["authToken"] KeyError: 'authToken'

az version: { "azure-cli": "2.53.1", "azure-cli-core": "2.53.1", "azure-cli-telemetry": "1.1.0", "extensions": { "bastion": "0.2.5", "ssh": "2.0.2" } }

[pwaring]

I had the same issue on Linux, it was resolved by granting the relevant permissions. However, Azure seems to take a very long time (more than an hour in my experience) to pick up the new permissions, so I continued to get the error message until the following day.

I agree with the original reporter than the error message could be improved - failing to catch an exception and then dumping a stack trace to the terminal makes it very difficult to work out what is wrong.

[dwicahyohphidayat]

@pwaring I am using role contributor, but still got the same error:

which permission is it exactly?

[AkechiShiro]

@dwicahyohphidayat @pwaring @bastionsuppgithub @yonzhan @cveld The error to me seems to not be related to one exact permission upstream on Azure, it seems to be randomly occurring on some Azure CLI versions (or if you try to open multiple ssh connection maybe ?), and on some other not happening at all.

1. Please provide the following information : az version output ? as well as extension version and which one are installed.
2. Installed on which Linux distro and by which means was it installed ? (Did you install via pacman/apt/nix/rpm)

Latest version up to date is 2.53.1, I believe, I did encounter this issue on 2.53.0 under ArchLinux (by having azure-cli installed with Nix despite having proper permission on Azure) and on Debian 12 as well. The fix was installing azure-cli from the extra repositories and upgrading extensions (ssh wasn't up-to-date, bastion was already)

Running onto this issue on Debian 12, I gave up using the Azure CLI because it's just plain garbage, sadly.

I'm getting a bit tired of the actual status of Bastion usage because it is costing a lot of money for something that is barely working at all, I'll try to push an Azure ticket upstream, I'll let everyone here know if that succeeds and if yes, what was the exact issue and how Microsoft magically solves it.

Basically, this software has no coherent behavior across Linux distros, it feels like unless I fix everything that is wrong with it with Nix, I can't make it work properly.

Btw, I'm a maintainer of the Nix package for azure-cli

[AkechiShiro]

In my case the error was just because of an invalid/typo in the target-resource-id, this error message is pretty unhelpful, it should be more specific, because it seems the API from Microsoft responds with proper exceptions, but these exceptions messages are not carried by the Azure CLI to the user.

[dwicahyohphidayat]

for me it was also because the --target-resource-id, the subscription was written as subscription's name not id (random string)

[joaomariocosta]

any news on this one?

[cr4sh0ut]

same issue on version "azure-cli": "2.60.0", but "azure-cli": "2.61.0", fixes it

[ilhaan]

I am running azure-cli 2.61.0 and seeing a similar issue. Verbose logging shows this:

Got a connection, starting a new thread
Exception in handling client: 'authToken'
Both debugger and websocket threads stopped...
Stopped local server..