for-mac icon indicating copy to clipboard operation
for-mac copied to clipboard
verified publisher icon docker

Cannot push image through an SSH tunnel

Open benjamingwynn opened this issue 2 years ago • 4 comments

Description

We're connecting to a remote registry over SSH using a tunnel to deploy to a remote network.

Docker push always prints the following:

Get "http://127.0.0.1:5000/v2/": dial tcp 127.0.0.1:5000: connect: connection refused

However I can clearly access the server just fine, as when I curl 127.0.0.1:5000/v2/, I get a valid result.

  • Tried with "Use Virtualization Framework" on/off as suggested in other issues, no difference.
  • I tried disabling "Airplay receiver" in system preferences, no difference.
  • I tried rebooting, no difference.

I do not think this is an IPv6 vs IPv4 problem, as I'm using ipv4 loopback specifically rather than localhost.

Works as expected on Linux.

Related: https://github.com/docker/for-mac/issues/6141, https://github.com/docker/for-mac/issues/6055

Reproduce

  1. ssh -NL 5000:127.0.0.1:5000 user@host
  2. docker pull hello-world
  3. docker tag hello-world 127.0.0.1:5000/hello-world
  4. docker push 127.0.0.1:5000/hello-world

Expected behavior

docker push should push

docker version

Client:
 Cloud integration: v1.0.33
 Version:           24.0.2
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        cb74dfc
 Built:             Thu May 25 21:51:16 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.20.1 (110738)
 Engine:
  Version:          24.0.2
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       659604f
  Built:            Thu May 25 21:50:59 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.2
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.5
    Path:     /Users/datacap/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     /Users/datacap/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/datacap/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.19
    Path:     /Users/datacap/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.4
    Path:     /Users/datacap/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/datacap/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/datacap/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  v0.12.0
    Path:     /Users/datacap/.docker/cli-plugins/docker-scout

Server:
 Containers: 36
  Running: 0
  Paused: 0
  Stopped: 36
 Images: 54
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.49-linuxkit-pr
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 2
 Total Memory: 2.921GiB
 Name: docker-desktop
 ID: 24c6dbbc-bc82-4560-93b5-54aebafaf5d6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  127.0.0.1:5000
  hubproxy.docker.internal:5555
  kubernetes.docker.internal:5000
  0.0.0.0:5000
  127.0.0.122:5000
  127.0.0.0/8
 Live Restore Enabled: false

Diagnostics ID

6F9316B5-F748-48D1-B2ED-A15F160469F2/20230608081429

Additional Info

  • Having to upload 145MB of potentially sensitive system information to generate a requried diagnostics ID seems unnecessary.
  • Automatically closing issues (like #6141) does not reduce the amount of issues, because users like me have to recreate a new issue to report the same problem.

benjamingwynn avatar Jun 08 '23 08:06 benjamingwynn

/lifecycle frozen

benjamingwynn avatar Jun 08 '23 08:06 benjamingwynn

Having the same issue here on Docker version 24.0.6-rd, build da4c87 for mac. Any updates?

== EDIT1 == Right after I asked this I've realized that docker desktop on MacOS runs inside of a Linuxkit VM and which has no access to the SSH tunnel running on the host. Still, curious how people are pushing images to remote internal registries from their Macs. My ugly solution for now:

docker save alpine:latest | ssh -C user@remote-internal-registry "docker load && docker tag alpine:latest localhost:5000/alpine:latest && docker push localhost:5000/alpine:latest" == EDIT2 == It is not that the docker VM does not have access to the tunnel on the host but that docker interprets "localhost" and "127.0.0.1" as the VM and not the host. The trick is to make the ssh tunnel available in all IPs on the host so you can point docker to something that is not localhost or 127.0.0.1. Using socat, for example:

socat TCP-LISTEN:5000,fork,reuseaddr TCP:127.0.0.1:5000

I can now point my docker push to another IP on my host and it will work:

❯ docker push 192.168.5.64:5000/ubuntu
Using default tag: latest
The push refers to repository [192.168.5.64:5000/ubuntu]
03fdf04efd9e: Pushing [==================================================>]  101.3MB
03fdf04efd9e: Pushed
latest: digest: sha256:781d090bf99d9b31a5a445ad2d42e516b5d677ca28638e4626f07d9df1e5cbf3 size: 529

...and that's because now we're listening on all ports (which may or may not be a security issue in your case depending on firewall rules, etc):

❯ netstat -vanp tcp | grep LISTEN | grep 5000
tcp46      0      0  *.5000                 *.*                    LISTEN       131072  131072  11019      0 00000 00000006 0000000003f68f90 00000000 00000800      1      0 000001
tcp4       0      0  127.0.0.1.5000         *.*                    LISTEN       131072  131072   5971      0 00100 00000006 0000000003f665ee 00000000 00000900      1      0 000001

laurentlgm avatar May 14 '24 18:05 laurentlgm

@laurentlgm that's a great workaround, however it seems quite dangerous and certainly not something I'd want in my deploy scripts.

Hopefully this can be resolved in Docker, but it doesn't seem like this is a huge priority for the docker team, sadly.

I ended up using a Linux VM in UTM to push Docker images to our remote docker hub via SSH - obviously not ideal either.

benjamingwynn avatar Jun 01 '24 01:06 benjamingwynn

I found that the /etc/hosts workaround in this issue https://github.com/docker/for-mac/issues/3611 works for this one as well. As in adding an alias for 127.0.0.1 to /etc/hosts like

127.0.0.1	      localhost registry.me

miikaah avatar Sep 14 '24 08:09 miikaah