bastion tunnel does not accept concurrent connections

[edwardsp]

az feedback auto-generates most of the information requested below, as of CLI version 2.0.62

Related command

az network bastion tunnel

Describe the bug

Only a single connection can be handled at one time with the bastion tunnel. All other attempts will block until the previous connection has closed.

This makes port forwarding for web applications unusable since the browser will typically have multiple connections open and only one will succeed.

To Reproduce

The easiest way to show this behaviour is to create a tunnel for ssh, e.g.

az network bastion tunnel --name bastion --resource-group $rg --target-resource-id $res_id --resource-port 22 --port 8822

Now, open two terminals and ssh to each, e.g.

ssh -p 8822 localhost

The second attempt will just hang and, if you close the first, it will connect.

Expected behavior

A tunnel should accept multiple connections.

Environment summary

Linux

Additional context

Here is a possible fix - https://github.com/edwardsp/azure-cli/commit/412a7198f6d2b4976552c2c0b4b3c33de35ef993 - although check the commit message as the threads are not joined and the cleanup is not called.

A workaround for creating a tunnel is to use az network bastion ssh. Once connected, through ssh, you can create a tunnel through that ssh connection. Type the three keys/characters: <Enter> <~> <C>. You will then be given the ssh> prompt where you can create a tunnel as you would with an ssh command line, e.g. for forwarding the resource port 8888 to local port 8888 you would do:

paul@azvm:~$
ssh> -L 8888:localhost:8888

paul@azvm:~$

[yonzhan]

route to CXP team

[ltalirz]

This is closely related to https://github.com/Azure/azure-cli/issues/22562

Since this renders the bastion tunnel feature unusable for many basic applications, and @edwardsp already identified a workaround above, it would be great if someone could have a look into getting this fixed.

cc @matt-chan

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @aznetsuppgithub.

Issue Details

---

az feedback auto-generates most of the information requested below, as of CLI version 2.0.62

Related command

az network bastion tunnel

Describe the bug

Only a single connection can be handled at one time with the bastion tunnel. All other attempts will block until the previous connection has closed.

This makes port forwarding for web applications unusable since the browser will typically have multiple connections open and only one will succeed.

To Reproduce

The easiest way to show this behaviour is to create a tunnel for ssh, e.g.

az network bastion tunnel --name bastion --resource-group $rg --target-resource-id $res_id --resource-port 22 --port 8822

Now, open two terminals and ssh to each, e.g.

ssh -p 8822 localhost

The second attempt will just hang and, if you close the first, it will connect.

Expected behavior

A tunnel should accept multiple connections.

Environment summary

Linux

Additional context

Here is a possible fix - https://github.com/edwardsp/azure-cli/commit/412a7198f6d2b4976552c2c0b4b3c33de35ef993 - although check the commit message as the threads are not joined and the cleanup is not called.

A workaround for creating a tunnel is to use az network bastion ssh. Once connected, through ssh, you can create a tunnel through that ssh connection. Type the three keys/characters: <Enter> <~> <C>. You will then be given the ssh> prompt where you can create a tunnel as you would with an ssh command line, e.g. for forwarding the resource port 8888 to local port 8888 you would do:

paul@azvm:~$
ssh> -L 8888:localhost:8888

paul@azvm:~$

Author:	edwardsp
Assignees:	-
Labels:	Network, Service Attention, Network - Bastion, CXP Attention, Auto-Assign
Milestone:	Backlog

[navba-MSFT]

@edwardsp Adding Service team to look into this.

@aznetsuppgithub Could you please look into this once you get a chance ?

[AkechiShiro]

Ran into this issue, again is there any progress on a fix ?

[AkechiShiro]

Hi @ltalirz @edwardsp,

As Microsoft doesn't seem to care much about this issue. Here is a quick workaround, I tested and it works.

This works-around the issue : https://stackoverflow.com/questions/72114013/howto-run-multiple-ssh-session-through-the-one-forwarded-port-port-redirectio It's called SSH Multiplexing : https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing#Advantages_of_Multiplexing

[tillahoffmann]

A workaround that has successfully worked for me is as follows.

1. Forward the ssh port 22 from the remote machine to a free port on the local machine (such as 2222).

az network bastion tunnel \
--name [name] \
--resource-group [resource-group] \
--target-resource-id [target-resource-id] \
--resource-port 22 \
--port 2222

2. Connect to the forwarded port and forward 22 again to another free port (such as 2223).

# Execute once.
ssh -p 2222 -L 2223:127.0.0.1:22 [user]@127.0.01

3. Connect an arbitrary number of sessions on port 2223, i.e.,

# Execute as many times as desired.
ssh -p 2223 [user]@127.0.0.1

It does however seem odd that such hoop-jumping is required for quite standard functionality.

[jebukhari]

A workaround that has successfully worked for me is as follows.

1. Forward the ssh port 22 from the remote machine to a free port on the local machine (such as 2222).

az network bastion tunnel \
--name [name] \
--resource-group [resource-group] \
--target-resource-id [target-resource-id] \
--resource-port 22 \
--port 2222

2. Connect to the forwarded port and forward 22 again to another free port (such as 2223).

# Execute once.
ssh -p 2222 -L 2223:127.0.0.1:22 [user]@127.0.01

3. Connect an arbitrary number of sessions on port 2223, i.e.,

# Execute as many times as desired.
ssh -p 2223 [user]@127.0.0.1

It does however seem odd that such hoop-jumping is required for quite standard functionality.

This works perfectly. The official instructions here did not work for me. It complained of port 22 being in use.

[AkechiShiro]

Still not answer from Microsoft ? @yonzhan ? @edwardsp @aznetsuppgithub

This issue is starting to take a lot of time to fix and the bastion costs a lot of money, I'm tired of workaround as they interfere with other tasks.

[tillahoffmann]

As a more comprehensive solution, also addressing the poor reliability of az bastion tunnel (cf. #28367), the following supervisord configuration has worked for me. The first program starts the tunnel and kills all azure commands because az network bastion tunnel sometimes just hangs without error messages. The second program does the "inception" tunneling. The whole thing will restart if the second program exits because az network bastion tunnel drops the connection. Pretty painful given that Bastion is a paid-for product.

[supervisord]
nodaemon=true

[program:az-bastion-network-tunnel]
startsecs = 5
startretries = 10
stdout_logfile = /dev/fd/1
stdout_logfile_maxbytes = 0
stderr_logfile = /dev/fd/1
stderr_logfile_maxbytes = 0
command = bash -c 'kill -9 `pgrep -f azure` || az network bastion tunnel \
    --name [name] \
    --resource-group [resource group] \
    --resource-port 22 \
    --port 2222 \
    --target-resource-id [target resource id]'

[program:inception]
startsecs = 5
startretries = 10
stdout_logfile = /dev/fd/1
stdout_logfile_maxbytes = 0
stderr_logfile = /dev/fd/1
stderr_logfile_maxbytes = 0
command = ssh -p 2222 -L [local port]:127.0.0.1:[remote port] ... [user]@127.0.0.1

[AkechiShiro]

@tillahoffmann Why use supervisord and not systemd ? Just curious if it's better for managing the azure tunnel ?

[tillahoffmann]

I used supervisord because I'm more familiar with it. systemd probably also does the trick.

[xpillons]

@aznetsuppgithub @navba-MSFT @yonzhan any news on this ? this is really painful issue that lot's of us are hitting.

[AkechiShiro]

@xpillons I recommend using other bastion such as OVH's bastion or using Sanzu it is less costy and work way better and is fully open source and also more responsive on bug reports.

But it needs management and a little bit of maintenance

[xpillons]

@AkechiShiro thanks for your answer but I want an Azure Native solution.

[AkechiShiro]

@xpillons you can pay for a support ticket at Microsoft they might answer you about this issue

[edwardsp]

This is now fixed in v1.0.1 of the bastion extension. Upgrade an existing installation with:

az extension update --name bastion 

[AkechiShiro]

Will try thanks