Increase max ssh connections on podman machine

[wbrefvem]

Feature request description

Currently, podman machine uses the default 10 max ssh connections, which causes errors in situations like the one described here: https://github.com/kubernetes-sigs/kind/issues/3742 .

Suggest potential solution

Manually increasing the max connections through MaxSessions and MaxStartups in an sshd config works, so automatically increasing it when creating the podman machine should be a straightforward approach.

[baude]

@jakecorrenti you were sort of most recent to mess with the SSH code, what do you think here?

[jakecorrenti]

If I'm understanding this correctly, it's a matter of changing the /etc/ssh/sshd_config file in the guest?

If this is something we want to support I could see it looking like podman machine init --max-ssh-sessions or podman machine set --max-ssh-sessions. We could even just have some documentation about how to do this manually.

I don't know enough about this to determine if there are any security or performance implications that we need to be concerned about.

[Luap99]

I don't think it must be configurable, given we only allow access from localhost anyway I see no real risk or performance problems here.

The question is more what limit do we want to set? 1000, more, less...?

I am not sure what the exact limit is set, I was able to run more than 10 podman-remote process concurrently but it started to fail when going towards 100 so I can reproduce but the question is what is reasonable?

[jakecorrenti]

I think answering "what is reasonable" is difficult. There's always going to be a user that isn't satisfied with the default we choose.

This is the first time I've seen an issue come up with this, so I'm leaning towards leaving the sshd config as is. We can always direct the user on how to change the sshd config or we can add some documentation

@Luap99 @baude wdyt

[Luap99]

This is the first time I've seen an issue come up with this, so I'm leaning towards leaving the sshd config as is. We can always direct the user on how to change the sshd config or we can add some documentation

Sure but there is always a first time for any problem/bug. I don't see any negatives by increasing the limits. From limited testing after a certain point connections are "randomly dropped" (i.e. there doesn't seem to be a fixed number after which is starts failing) which is very hard for users to debug or understand and that just leads to a lot of wasted debugging time on all sides (even for us when they report bugs).

As such I think having a higher limit is a positive thing. Another alternative could be to switch the remote client with machine over to use the exposed API unix socket and use a ssh connection per cli command.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[BenTheElder]

I expect even just 10 => 100 would help kind users (e.g. podman desktop?) with multi-node clusters.

We haven't encountered noticeable concurrency limits on container commands before except in this case, since the limit isn't easily discoverable and seems safe to increase, we don't really want to impose an arbitrarily low limit on all of our users.

In our case we're trying to quickly collect log files from the containers for debugging, this is normally a very cheap operation that we can do with high concurrency.

[Luap99]

This was fixed in https://github.com/containers/podman-machine-os/pull/84 which should be part of the 5.4 image I belive

[BenTheElder]

Thank you!