Issue after update to Docker CE 20.10.13 / Containerd (1.5.10)

[fhaefemeier]

After update the host with latest docker packages (20.10.13) containing docker-ce, docker-ce-cli, docker-ce-rootless-extras and containerd-io (1.5.10) sftp connections are not fully established anymore. The log (Loglevel: verbose) print:

player_1   | Connection from 192.168.yyy.xxx port 44762 on 10.10.yyy.xxx port 22
player_1   | Postponed publickey for player from 192.168.yyy.xxx port 44762 ssh2 [preauth]
player_1   | Accepted publickey for player from 192.168.yyy.xxx port 44762 ssh2: RSA SHA256:0ZshKyTwjB/ac6n0EdIxkNYT8RfvNRZONdqUEWA4YSQ
player_1   | User child is on pid 33
player_1   | Changed root directory to "/home/player"
player_1   | Starting session: forced-command (config) 'internal-sftp' for player from 192.168.yyy.xxx port 44762 id 0

But the client does not receive a 'Connected' message and both sides are waiting. The behaviour can be reproduced remote (different machines), local on the host (via NAT/localhost) and inside the sftp container (with docker exec).

After downgrade all packages back to previous version docker 20.10.12 and containerd (1.4.13) the issue is gone and the sftp service is working as expected. No other changes (e.g. network settings, firewall, ...) between update/downgrade had happen.

[fhaefemeier]

Any news? The problem is still happen. I upgrade yesterday to docker from 20.10.12 to 20.10.16 and containerd from 1.4.13 to 1.6.4. How can I support you?

[imsandli]

I'm facing the same issue with Alma 9 (containerd.io-1.6.6-3.1.el9.x86_64)

eventually it connected ~14 Minutes after entering the password.

Docker run command

docker run -d atmoz/sftp foo:pass:::upload
sftp foo@silly_lovelace 
foo@silly_lovelace's password: 
*** Enter Password and wait ~14 Minutes ***
Connected to silly_lovelace.
sftp> 

Docker Logs with debug level:

[/entrypoint] Executing sshd
Server listening on 0.0.0.0 port 22.
Server listening on :: port 22.
Connection from 172.18.0.1 port 36858 on 172.18.0.3 port 22 rdomain ""
Accepted password for foo from 172.18.0.1 port 36858 ssh2
User child is on pid 9
Changed root directory to "/home/foo"
Starting session: forced-command (config) 'internal-sftp' for foo from 172.18.0.1 port 36858 id 0
Received disconnect from 172.18.0.1 port 36858:11: disconnected by user
Disconnected from user foo 172.18.0.1 port 36858

[MamaTierra]

Same issue here. Blocked after entering the password even if the host has accepted the password:

[/entrypoint] Executing sshd
Server listening on 0.0.0.0 port 22.
Server listening on :: port 22.
Accepted password for *** from 192.168.2.21 port 48548 ssh2

[waweber]

I did some experimenting and I think the root cause is that the internal-sftp implementation requires the container to have CAP_SYS_ADMIN for something it does. It also requires /proc to be available when chrooted. Not sure what changed or what exactly it does, but the "old" sftp implementation does not have this issue.

My sshd_config has:

Subsystem sftp /usr/lib/openssh/sftp-server
ForceCommand /usr/lib/openssh/sftp-server
# And remove ChrootDirectory

And it works again. Note that chroot will require extra effort to set up because of the required files/directories.

[Lustyn]

I've found that SYS_ADMIN or changing sftp implementation is not necessary, there is an issue with the new default ulimits for file descriptors in newer docker versions. Having that value too high will eat CPU when establishing connections causing them to stall for several minutes.

See here for the fix: https://github.com/atmoz/sftp/issues/341#issuecomment-1509470933