docker
Docker starts, but there is no connection until docker.socket restarts
- [x] This is not a bug report
- [x] I searched existing issues before opening this one
I made an assembly based on Thinstation. Added package docker. I run the distribution, everything loads without problems. Docker starts after downloading the distribution. I download nginx:alpine and run it with port 80 forwarding. The container starts, there are no problems:
ts_0800273e9027:~# docker run --restart always --name mynginx -p 80:80 -d nginx:alpine
4fc242d58285: Loading layer [==================================================>] 5.855MB/5.855MB
4721bfafc708: Loading layer [==================================================>] 19.05MB/19.05MB
45b275e8a06d: Loading layer [==================================================>] 3.072kB/3.072kB
a43749efe4ec: Loading layer [==================================================>] 4.096kB/4.096kB
d6dd885da0bb: Loading layer [==================================================>] 3.584kB/3.584kB
c0e7c94aefd8: Loading layer [==================================================>] 7.168kB/7.168kB
Loaded image: nginx:alpine
a22e21817fbd4040e6cbf0d590180117ee88cecb1259080e7af4016b0931b5ce
ts_0800273e9027:~# docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a22e21817fbd nginx:alpine "/docker-entrypoint.…" 10 seconds ago Up 9 seconds 0.0.0.0:80->80/tcp, :::80->80/tcp mynginx
ts_0800273e9027:~# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 3678/docker-proxy
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 2321/dropbear
tcp 0 0 :::80 :::* LISTEN 3685/docker-proxy
tcp 0 0 :::22 :::* LISTEN 2321/dropbear
Rules table:
ts_0800273e9027:~# iptables -L -v -n
Chain INPUT (policy ACCEPT 1064 packets, 77952 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 963 packets, 651K bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp -- !docker0 docker0 0.0.0.0/0 172.17.0.2 tcp dpt:80
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
0 0 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
The problem is that I can't connect even locally to the lifted container at the localhost address. Telnet also does not connect to port 80. The remote connection doesn't work either.
If I restart docker.socket - access to the container from the outside is provided, I open the page in the browser.
My question is this: how to understand and why the initial launch of Docker after downloading the distribution does not provide access to running containers? Why does it start working only after restarting docker.socket?
I also attach the full distribution download log and the Docker startup log.
I also provide an infographic of the distribution download:
UPD. I have reassembled the assembly with the ability to debug (/usr/bin/dockerd --debug -H fd://). I attach the FULL docker download docker.log.
I have extracted the log BEFORE and AFTER restarting docker.socket. I share a comparison of two logs. As you can see, after restarting docker.socket adds new rules via iptables.
I'm getting a similar issue. In my case the container runs well but if I reboot my VM my container is show by docker ps and the logs show that is alright but when I tried to access on browsers it's fails.
I can only resolve restarting my container.
I'm years late looking at this - and I guess everything will have changed since. But, fwiw ...
The bulk of the extra rules in the log diff are a red herring. The default bridge gets created for the first time in the initial run of the daemon. After the restart, it's created by restoration from the store, then re-created because it's always deleted on startup (unless it's a "live-restore").
So, I think the important line in the startup log diff is probably:
level=debug msg="/sbin/iptables, [--wait -t filter -P FORWARD DROP]"
In its initial startup, Docker set the iptables filter-FORWARD policy to DROP. In the restart, because it didn't enable sysctl ip_forward, it didn't set the default policy.
But, for that to make a difference, something would need to be resetting the policy during the Docker restart, but I don't know what that might be. And, there must be something interesting about the networking the browser is using to reach Docker ... but I don't understand enough about Thinstation to know what that might be. So, the theory is weak! It'd be good to understand where the browser and docker are running. (There's some more info at https://github.com/Thinstation/thinstation/issues/724, but I'm still unclear on how everything fits together.)
If it's still a current issue, or anyone else hits the same problem - running iptables -P FORWARD ACCEPT (where Docker is running, after the initial startup) would narrow things down. If it still doesn't work after that, my theory is wrong!
In the upcoming moby 28.0 (release candidates are here), the iptables rules have changed a bit. There's no longer a dependency on the FORWARD chain's default policy. That may mean the restart workaround here stops working ... but I'm not sure, it depends on what exactly was going wrong. So in 28.0, it might be necessary to create a user-defined network (instead of the default bridge) with option -o com.docker.network.bridge.gateway_mode_ipv4=nat-unprotected.
