desktop-linux
desktop-linux copied to clipboard
docker
Can not list all ports the system LISTEN to when docker daemon crashed
Description
I had not visited my host about 1 year. There is SSHD daemon and docker with different containers which expose their ports. This is Ubuntu system. Docker service seems is down.
Reproduce
# uname -a
Linux ubuntu-2gb-nbg1-2 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
# netstat -na
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:52697 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp6 0 0 ::1:52697 :::* LISTEN
tcp6 0 0 :::22 :::* LISTEN
udp 0 0 127.0.0.53:53 0.0.0.0:*
udp 0 0 myip:68 0.0.0.0:*
udp 0 0 0.0.0.0:4789 0.0.0.0:*
raw6 0 0 :::58 :::* 7
Active UNIX domain sockets (servers and established)
# docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
# systemctl status docker
× docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2023-12-21 05:41:24 UTC; 1 week 5 days >
TriggeredBy: × docker.socket
Docs: https://docs.docker.com
Main PID: 3068220 (code=exited, status=1/FAILURE)
CPU: 96ms
Notice: journal has been rotated since unit was started, output may be incomplete.
# docker --version
Docker version 20.10.21, build baeda1f
$ curl -k https://myip:9443
<!doctype html>...
But I know that some containers are still running, because I can open a VPN connection and access to https://myip:9443.
Why I do not see this :9443 port from netstat -na output?
UPD
I cleanup the space: https://superuser.com/a/1824057/431840
After starting docker service back I can see invisible services:
# netstat -na
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:9443 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:52697 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN
tcp6 0 0 ::1:52697 :::* LISTEN
tcp6 0 0 :::9443 :::* LISTEN
tcp6 0 0 :::22 :::* LISTEN
tcp6 0 0 :::80 :::* LISTEN
tcp6 0 0 :::443 :::* LISTEN
udp 0 0 127.0.0.53:53 0.0.0.0:*
udp 0 0 myip:68 0.0.0.0:*
udp 0 0 0.0.0.0:4789 0.0.0.0:*
raw6 0 0 :::58 :::* 7
Expected behavior
If I can access 9443 port (exposed from portainer container), then system listens on 9443 port and this should be displayed regrading docker service is run or not.
docker version
# docker version
Client: Docker Engine - Community
Version: 20.10.21
API version: 1.41
Go version: go1.18.7
Git commit: baeda1f
Built: Tue Oct 25 18:01:58 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.21
API version: 1.41 (minimum version 1.12)
Go version: go1.18.7
Git commit: 3056208
Built: Tue Oct 25 17:59:49 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.9
GitCommit: 1c90a442489720eec95342e1789ee8a5e1b9536f
runc:
Version: 1.1.4
GitCommit: v1.1.4-0-g5fd4c4d
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
# docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
compose: Docker Compose (Docker Inc., v2.12.2)
scan: Docker Scan (Docker Inc., v0.21.0)
Server:
Containers: 30
Running: 8
Paused: 0
Stopped: 22
Images: 39
Server Version: 20.10.21
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: local
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: error
NodeID:
Error: error while loading TLS certificate in /mnt/docker-overlay/swarm/certificates/swarm-node.crt: certificate (1 - 38oh62w7k7zugmteg0xyzqn9g) not valid after Wed, 27 Dec 2023 01:39:00 UTC, and it is currently Tue, 02 Jan 2024 22:16:41 UTC: x509: certificate has expired or is not yet valid:
Is Manager: false
Node Address: 167.235.58.111
Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 1c90a442489720eec95342e1789ee8a5e1b9536f
runc version: v1.1.4-0-g5fd4c4d
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
cgroupns
Kernel Version: 5.15.0-56-generic
Operating System: Ubuntu 22.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.578GiB
Name: ubuntu-2gb-nbg1-2
ID: X6IR:CQPM:NB3G:ZGBT:WWGU:UIC3:UDD4:TPNV:GHFF:4GRR:YVKC:OUMF
Docker Root Dir: /mnt/docker-overlay
Debug Mode: true
File Descriptors: 94
Goroutines: 97
System Time: 2024-01-02T22:32:20.341046307Z
EventsListeners: 2
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Diagnostics ID
no id
Additional Info
I suppose docker was crushed because of storage issue. Do not know why it takes so much space. I configured log rotation:
# ls -la
total 6818696
drwx--x--- 4 root root 4096 Jan 2 22:29 .
drwx--x--- 33 root root 4096 Jan 26 2023 ..
-rw-r----- 1 root root 6978717361 Jan 2 22:29 a90baaa15da31f2ab9a4683af417b232166e7bf84591a1cd4f96aed858f06e8a-json.log
drwx------ 2 root root 4096 Jan 16 2023 checkpoints
-rw------- 1 root root 3661 Jan 2 22:29 config.v2.json
-rw-r--r-- 1 root root 1485 Jan 2 22:29 hostconfig.json
-rw-r--r-- 1 root root 13 Jan 2 22:29 hostname
-rw-r--r-- 1 root root 150 Jan 2 22:29 hosts
drwx--x--- 2 root root 4096 Jan 16 2023 mounts
-rw-r--r-- 1 root root 53 Jan 2 22:29 resolv.conf
-rw-r--r-- 1 root root 71 Jan 2 22:29 resolv.conf.hash
root@ubuntu-2gb-nbg1-2:/mnt/docker-overlay/containers/a90baaa15da31f2ab9a4683af417b232166e7bf84591a1cd4f96aed858f06e8a# cat /etc/docker/daemon.json
{
"debug": true,
"data-root": "/mnt/docker-overlay",
"features":
{ "buildkit": true },
"log-driver": "local",
"log-opts": {
"max-size": "500m",
"max-file": "3"
}
}
Latest logs from portainer service (port 9443):
{"time":1703948774,"message":"http: TLS handshake error from 87.236.176.41:41127: EOF"}
{"time":1703948807,"message":"http: TLS handshake error from 87.236.176.43:37637: EOF"}
{"time":1703950290,"message":"http: TLS handshake error from 185.233.19.154:38080: EOF"}
{"time":1703950290,"message":"http: TLS handshake error from 185.233.19.154:38158: EOF"}
{"time":1703950290,"message":"http: TLS handshake error from 185.233.19.154:38328: tls: no cipher suite supported by both client and server"}
{"time":1703950291,"message":"http: TLS handshake error from 185.233.19.154:38440: tls: client requested unsupported application protocols ([http/0.9 http/1.0 spdy/1 spdy/2 spdy/3 h2c hq])"}
{"time":1703950291,"message":"http: TLS handshake error from 185.233.19.154:38862: tls: client requested unsupported application protocols ([hq h2c spdy/3 spdy/2 spdy/1 http/1.0 http/0.9])"}
{"time":1703950291,"message":"http: TLS handshake error from 185.233.19.154:39022: tls: client offered only unsupported versions: [302 301]"}
{"time":1703950292,"message":"http: TLS handshake error from 185.233.19.154:39136: EOF"}
{"time":1703950292,"message":"http: TLS handshake error from 185.233.19.154:39376: EOF"}
{"time":1703950293,"message":"http: TLS handshake error from 185.233.19.154:39628: EOF"}
{"time":1703950293,"message":"http: TLS handshake error from 185.233.19.154:40084: EOF"}
{"time":1703950294,"message":"http: TLS handshake error from 185.233.19.154:40328: EOF"}
{"time":1703951039,"message":"http: TLS handshake error from 80.66.88.204:65062: tls: first record does not look like a TLS handshake"}
{"time":1703951155,"message":"http: TLS handshake error from 165.154.244.17:47702: EOF"}
{"time":1703951155,"message":"http: TLS handshake error from 165.154.244.17:50186: tls: first record does not look like a TLS handshake"}
{"time":1703951175,"message":"http: TLS handshake error from 165.154.244.17:54624: EOF"}
{"time":1703951176,"message":"http: TLS handshake error from 165.154.244.17:54852: EOF"}
{"time":1703951177,"message":"http: TLS handshake error from 165.154.244.17:55080: tls: no cipher suite supported by both client and server"}
{"time":1703951178,"message":"http: TLS handshake error from 165.154.244.17:55360: tls: client requested unsupported application protocols ([http/0.9 http/1.0 spdy/1 spdy/2 spdy/3 h2c hq])"}
{"time":1703951178,"message":"http: TLS handshake error from 165.154.244.17:55462: tls: client requested unsupported application protocols ([hq h2c spdy/3 spdy/2 spdy/1 http/1.0 http/0.9])"}
{"time":1703951180,"message":"http: TLS handshake error from 165.154.244.17:55558: tls: client offered only unsupported versions: [302 301]"}
{"time":1703951181,"message":"http: TLS handshake error from 165.154.244.17:55860: EOF"}
{"time":1703951182,"message":"http: TLS handshake error from 165.154.244.17:56074: EOF"}
{"time":1703951183,"message":"http: TLS handshake error from 165.154.244.17:56270: EOF"}
{"time":1703951184,"message":"http: TLS handshake error from 165.154.244.17:56438: EOF"}
{"time":1703977018,"message":"http: TLS handshake error from 205.210.31.17:50013: tls: client offered only unsupported versions: [302 301]"}
{"time":1703998157,"message":"http: TLS handshake error from 162.243.152.18:39820: tls: first record does not look like a TLS handshake"}
{"time":1704040224,"message":"http: TLS handshake error from 185.233.19.145:38628: EOF"}
{"time":1704040225,"message":"http: TLS handshake error from 185.233.19.145:38722: EOF"}
{"time":1704040225,"message":"http: TLS handshake error from 185.233.19.145:39130: tls: no cipher suite supported by both client and server"}
{"time":1704040225,"message":"http: TLS handshake error from 185.233.19.145:39190: tls: client requested unsupported application protocols ([http/0.9 http/1.0 spdy/1 spdy/2 spdy/3 h2c hq])"}
{"time":1704040226,"message":"http: TLS handshake error from 185.233.19.145:39256: tls: client requested unsupported application protocols ([hq h2c spdy/3 spdy/2 spdy/1 http/1.0 http/0.9])"}
{"time":1704040226,"message":"http: TLS handshake error from 185.233.19.145:39280: tls: client offered only unsupported versions: [302 301]"}
{"time":1704040227,"message":"http: TLS handshake error from 185.233.19.145:39306: EOF"}
{"time":1704040227,"message":"http: TLS handshake error from 185.233.19.145:39380: EOF"}
{"time":1704040227,"message":"http: TLS handshake error from 185.233.19.145:39406: EOF"}
{"time":1704040228,"message":"http: TLS handshake error from 185.233.19.145:39428: EOF"}
{"time":1704040228,"message":"http: TLS handshake error from 185.233.19.145:39524: EOF"}
{"time":1704043958,"message":"http: TLS handshake error from 183.136.225.42:55640: EOF"}
{"time":1704043958,"message":"http: TLS handshake error from 183.136.225.42:42830: EOF"}
{"time":1704043959,"message":"http: TLS handshake error from 183.136.225.42:5875: tls: no cipher suite supported by both client and server"}
{"time":1704043959,"message":"http: TLS handshake error from 183.136.225.42:49147: tls: client requested unsupported application protocols ([http/0.9 http/1.0 spdy/1 spdy/2 spdy/3 h2c hq])"}
{"time":1704043960,"message":"http: TLS handshake error from 183.136.225.42:63690: tls: client requested unsupported application protocols ([hq h2c spdy/3 spdy/2 spdy/1 http/1.0 http/0.9])"}
{"time":1704043960,"message":"http: TLS handshake error from 183.136.225.42:37466: tls: client offered only unsupported versions: [302 301]"}
{"time":1704043961,"message":"http: TLS handshake error from 183.136.225.42:3462: EOF"}
{"time":1704043961,"message":"http: TLS handshake error from 183.136.225.42:12900: EOF"}
{"time":1704043962,"message":"http: TLS handshake error from 183.136.225.42:49272: EOF"}
{"time":1704043963,"message":"http: TLS handshake error from 183.136.225.42:59873: EOF"}
{"time":1704044947,"message":"http: TLS handshake error from 80.66.88.215:65340: tls: first record does not look like a TLS handshake"}
{"time":1704046636,"message":"http: TLS handshake error from 117.13.169.92:20792: EOF"}
{"time":1704069953,"message":"http: TLS handshake error from 94.102.61.25:56162: EOF"}
{"time":1704085903,"message":"http: TLS handshake error from 198.235.24.127:55872: tls: client offered only unsupported versions: [302 301]"}
{"time":1704103441,"message":"http: TLS handshake error from 198.199.116.114:43932: tls: first record does not look like a TLS handshake"}
{"time":1704136910,"message":"http: TLS handshake error from 45.227.254.8:65392: tls: first record does not look like a TLS handshake"}
{"time":1704142048,"message":"http: TLS handshake error from 185.170.144.3:64932: tls: first record does not look like a TLS handshake"}
{"time":1704165499,"message":"http: TLS handshake error from 205.210.31.231:50846: tls: client offered only unsupported versions: [302 301]"}
{"time":1704205688,"message":"http: TLS handshake error from 87.236.176.168:44027: EOF"}
{"time":1704205721,"message":"http: TLS handshake error from 87.236.176.170:59003: EOF"}
{"time":1704205754,"message":"http: TLS handshake error from 87.236.176.149:37635: tls: no cipher suite supported by both client and server"}
{"time":1704205787,"message":"http: TLS handshake error from 87.236.176.162:58875: tls: client requested unsupported application protocols ([http/0.9 http/1.0 spdy/1 spdy/2 spdy/3 h2c hq])"}
{"time":1704205820,"message":"http: TLS handshake error from 87.236.176.159:43167: tls: client requested unsupported application protocols ([hq h2c spdy/3 spdy/2 spdy/1 http/1.0 http/0.9])"}
{"time":1704205853,"message":"http: TLS handshake error from 87.236.176.169:33443: tls: client offered only unsupported versions: [302 301]"}
{"time":1704205886,"message":"http: TLS handshake error from 87.236.176.171:32789: EOF"}
{"time":1704205919,"message":"http: TLS handshake error from 87.236.176.176:45919: EOF"}
{"time":1704205952,"message":"http: TLS handshake error from 87.236.176.177:48709: EOF"}
{"time":1704205985,"message":"http: TLS handshake error from 87.236.176.176:48585: EOF"}
{"time":1704226521,"message":"http: TLS handshake error from 104.158.164.191:52286: local error: tls: bad record MAC"}
{"time":1704228486,"message":"http: TLS handshake error from 159.203.192.10:39954: tls: first record does not look like a TLS handshake"}
{"time":1704232286,"message":"http: TLS handshake error from 45.227.254.48:65434: tls: first record does not look like a TLS handshake"}
I found why disk space was consumed: I need to rebuild my containers. https://superuser.com/a/1824057/431840
It would be nice to have some tool which will allow to find containers which need to be reconfigured to use the current docker configuration.