1.28.5 issues connecting over SSH

[PhilJay]

Description of the issue

I've recently updated from docker-compose 1.27.4 to 1.28.5. Prior to the update, the following command worked as expected: docker-compose -f docker-compose.yml -H "ssh://[email protected]" up --build --force-recreate -d

After the update, I'm getting a connection refused error: ssh: connect to host domain.com port 22: Connection refused

Downgrading to 1.27.4 resolves the issue.

Context information

I am deploying to a linux server with ufw enabled, the configuration is as follows:

To                         Action      From
--                         ------      ----
22/tcp                     LIMIT       Anywhere                  
2375/tcp                   ALLOW       Anywhere                  
2376/tcp                   ALLOW       Anywhere                  
Apache Full                ALLOW       Anywhere                  
22/tcp (v6)                LIMIT       Anywhere (v6)             
2375/tcp (v6)              ALLOW       Anywhere (v6)             
2376/tcp (v6)              ALLOW       Anywhere (v6)             
Apache Full (v6)           ALLOW       Anywhere (v6)  

Output of docker-compose version

docker-compose version 1.28.5, build c4eb3a1f
docker-py version: 4.4.4
CPython version: 3.9.0
OpenSSL version: OpenSSL 1.1.1h  22 Sep 2020

Output of docker version

Client: Docker Engine - Community
 Cloud integration: 1.0.9
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:13:00 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:15:47 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker-compose config (Make sure to add the relevant -f and other flags)

networks:
  rmd:
    driver: bridge
services:
  rmd-cms:
    build:
      context: /project
      dockerfile: Dockerfile
    container_name: rmd-cms
    image: rmd-cms
    networks:
      rmd: null
    ports:
    - published: 1343
      target: 80
    restart: always
version: '3.7'

Steps to reproduce the issue

1. Install docker-compose 1.28.5
2. Setup linux server and enable ufw with the above rules
3. Use docker-compose up with SSH

Observed result

ssh: connect to host domain.com port 22: Connection refused

Expected result

docker-compose works as expected without a connection refused error.

[aiordache]

Hi @PhilJay. Is it working when you manually ssh to the host with ssh user@hostdomain?

To switch back to using the implementation from 1.27.4, you ca try setting COMPOSE_PARAMIKO_SSH=1 with 1.28.5

[PhilJay]

Yes, manual SSH is working as expected. Using COMPOSE_PARAMIKO_SSH=1 resolves the problem and gets rid of the connection refused error 👍 . Any idea what is different in 1.28.5 to cause this? Additionally, do you have any idea if this is related to #8222 ?

[aiordache]

@PhilJay In 1.28 we switched to shelling out to the SSH client to align it with the docker cli. We left the paramiko client as a fallback for cases when the shelling out does not work.

The new implementation spawns a python subprocess that executes a docker cli command through SSH to connect to the daemon. It runs ssh -l user -p port -- host docker system dial-stdio which allows to make HTTP queries via the stdin/stdout of the subprocess.

Would you mind trying to run this:

$ ssh -l user -p port -- hostdomain docker system dial-stdio
GET /v1.41/containers/json HTTP/1.1
Host: localhost
User-Agent: docker-compose/1.29.0dev
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive

Once you input 2 newlines you should get the container data from the daemon.

Do you always get the SSH error for all commands or is it only when you run with build?

[PhilJay]

@aiordache It appears that it's unrelated to build and occurs on every command I tried. The above command (updated with my user info of course) gives me a 400 error.

[aiordache]

@PhilJay Are you sure you you wrote the HTTP request as:

GET /v1.41/containers/json HTTP/1.1
Host: localhost
User-Agent: docker-compose/1.29.0dev
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive

You would get 400 if one of the fields is missing or is invalid:

$ ssh hostdomain docker system dial-stdio
GET /v1.41/containers/json HTTP/1.1

HTTP/1.1 400 Bad Request: missing required Host header
Content-Type: text/plain; charset=utf-8
Connection: close

400 Bad Request: missing required Host header

Do you get the 400 error codes for all queries? What is the error message ?

GET /version HTTP/1.1 or GET /v1.41/images/json HTTP/1.1

Are the docker cli commands working when you target the remote host:

$ docker -H ssh://user@hostdomain ps

[PhilJay]

@aiordache Ok got it now, was missing something.

I'm getting a 200 now for the request you wanted me to test, so it is working.

Concerning the docker cli commands targeting the remote host: They are working as well.

[aiordache]

If the manual ssh queries work fine, it should work with docker-compose too. It executes the same commands.

One last question to see if we can narrow down the issue. Did you use so far the 1.28.5 binary? Do you mind trying the 1.28.5 installed via pip. You can run a python virtualenv to test.

In case the one installed via pip works fine, the issue may be in the binary runtime setup with Pyinstaller.

[PhilJay]

I've now installed docker-compose via pip, tried the above mentioned command without using COMPOSE_PARAMIKO_SSH=1: Getting the same error: ssh: connect to host domain.com port 22: Connection refused

[quentinsf]

In case it's relevant, I was having a problem with 1.28.5 on my Mac, where every command was taking at least 15 secs. I was using SSH to a remote engine on Ubuntu, and had the ssh info set up as a docker context. The docker client was fine; this was only an issue with docker-compose. e.g.

docker-compose -c mycontext ps takes 15s docker -c mycontext ps takes 0.9s

Reverting to 1.27 or setting COMPOSE_PARAMIKO_SSH=1 made everything fast again in docker-compose, but I started hitting the errors in #6463 and had to increase MaxSessions on my remote sshd.

With the combinations of those two workarounds, it's fine.

1.28 in the non-paramiko mode is still painfully slow. (But everything does eventually work).

[jswny]

I'm also having the same problems using compose on a remote machine via SSH, have seen this with logs and down.

[jayjootar]

Had the same problem. Try using

docker compose up

instead of

docker-compose up

then, everything works. Though it says that docker compose up is still experimental.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[crshumate]

As I previously mentioned on the closed issue #6336 this is an issue in Travis-CI as well. Just wanted to pass along my experience on an open ticket.

I noticed this issue using docker-compose v 1.28.5 in Travis-CI.

I downgraded to 1.27.4 which is what we use in Travis-CI for another project and it worked as expected.

[stale[bot]]

This issue has been automatically marked as not stale anymore due to the recent activity.

[ndeloof]

docker <space> compose (compose v2) has been released in September, you should consider updating

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it had not recent activity during the stale period.