buildx over ssh no longer works with docker compose

[trajano]

Not sure if something has changed, but in Docker Desktop 4.30.0 windows docker compose yields

error during connect: Get "http://docker.example.com/v1.39/containers/buildx_buildkit_dit0/json": command [ssh -o ConnectTimeout=30 -- dit docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=Connection closed by xxxxx port 22

but docker buildx build still works.

Originally posted by @trajano in https://github.com/moby/buildkit/issues/4268#issuecomment-2147682560

Looking at the server logs under /var/secure I see when using with docker compose

sshd[###]: Connection reset by xx.xx.xx.xx port ### [preauth]

SSH with a normal client still works though.

I have also tried regenerating a new SSH key pair using the default ssh-keygen values and confirmed it is the one being used when I ssh directly. No luck.

[tonistiigi]

cc @ndeloof Any ideas?

[ndeloof]

I wonder this is caused by parallelism used by compose to orchestrate service images build (required to support cross-services base image references). We get a comparable issue reported running compose commands over ssh, where parallel ssh connexions are required per API calls. Please check https://github.com/docker/compose/issues/10117#issuecomment-1842261479 and see if same fix applies to this issue

[trajano]

I wonder this is caused by parallelism used by compose to orchestrate service images build (required to support cross-services base image references). We get a comparable issue reported running compose commands over ssh, where parallel ssh connexions are required per API calls. Please check docker/compose#10117 (comment) and see if same fix applies to this issue

Multiplexing does not work from Windows OpenSSH client since it needs sockets. Tried out https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing#Setting_Up_Multiplexing

I took the Path portion out and it went through but it disables the multiplexing from what I can tell.

https://github.com/docker/compose/issues/10117#issuecomment-1881197799 actually states it has issues with Windows

https://github.com/PowerShell/Win32-OpenSSH/issues/405 confirms that it is not implemented on Windows.

[trajano]

Not sure if it is buildx (which has it's own stdio) or part of Docker SDK but perhaps it should do connection pooling of SSH rather that start new ones and let the pool handle the retries.