moby
moby copied to clipboard
moby
Ubuntu 21.04: to bridge docker0 failed: could not find bridge docker0: route ip+net: no such network interface.
After a docker service starts, I can execute only one container. Restarting the container or starting a new one throws the following error:
$ sudo docker run --rm -ti ubuntu bash
docker: Error response from daemon: failed to create endpoint modest_turing on network bridge: adding interface veth620572c to bridge docker0 failed: could not find bridge docker0: route ip+net: no such network interface.
If I restart the service with sudo service docker restart and execute sudo docker run --rm -ti ubuntu bash again, it works.
Here are the docker daemon logs when the fail happens:
un 23 08:29:21 gandalf dockerd[3233]: time="2021-06-23T08:29:21.205486316-04:00" level=warning msg="Failed to delete host side interface (veth171e122)'s link" error="no such device"
Jun 23 08:29:21 gandalf dockerd[3233]: time="2021-06-23T08:29:21.215681107-04:00" level=error msg="82f07d51679aa97552e3c9950fae2ceb9210c12ca68a1597c3c797df79b7c50d cleanup: failed to delete container from containerd: no such container"
Jun 23 08:29:21 gandalf dockerd[3233]: time="2021-06-23T08:29:21.220122199-04:00" level=error msg="Handler for POST /v1.41/containers/82f07d51679aa97552e3c9950fae2ceb9210c12ca68a1597c3c797df79b7c50d/start returned error: failed to create endpoint quizzical_keldysh on network bridge: adding interface veth171e122 to bridge docker0 failed: could not find bridge docker0: route ip+net: no such network interface"
This bug started to happen when I upgraded from Ubuntu server 20.10 to 21.04. Note that I do have a TLS docker configuration setup but I have disabled everything to narrow down the bug (for example I don't use an /etc/docker/daemon.json file neither override the systemd service file).
Here are the versions and infos:
$ sudo docker version
Client: Docker Engine - Community
Version: 20.10.7
API version: 1.41
Go version: go1.13.15
Git commit: f0df350
Built: Wed Jun 2 12:00:45 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.7
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: b0f5bc3
Built: Wed Jun 2 11:58:56 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.6
GitCommit: d71fcd7d8303cbf684402823e425e9dd2e99285d
runc:
Version: 1.0.0-rc95
GitCommit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
docker-init:
Version: 0.19.0
GitCommit: de40ad0
$ sudo docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
scan: Docker Scan (Docker Inc., v0.8.0)
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 2
Server Version: 20.10.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.11.0-22-generic
Operating System: Ubuntu 21.04
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.55GiB
Name: gandalf
ID: 4EZV:Z6PX:LAOA:H3GD:VCPP:LYF3:ZGMN:HGK4:JG4E:TLYN:LZOG:KJCR
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp38s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 70:85:c2:fb:2c:73 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.192/24 brd 192.168.0.255 scope global enp38s0
valid_lft forever preferred_lft forever
inet6 fe80::7285:c2ff:fefb:2c73/64 scope link
valid_lft forever preferred_lft forever
3: wlp37s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 50:e0:85:f4:eb:47 brd ff:ff:ff:ff:ff:ff
Note that I am using the official Docker channel for the packages but I have also tried the one shipped by default in Ubuntu (currently 20.10.2) and see the same bug.
I have also looked into the closed and open issues of the repo without success.
It looks like it might be relevant to show the system logs as well when the error happens:
$ tail -f /var/log/syslog
Jun 23 09:06:08 gandalf systemd[1548]: var-lib-docker-overlay2-158dbf596177fea1dbaf78a255fea9366e73a17c99031f313203847b47d264c7\x2dinit-merged.mount: Succeeded.
Jun 23 09:06:08 gandalf systemd[1386]: var-lib-docker-overlay2-158dbf596177fea1dbaf78a255fea9366e73a17c99031f313203847b47d264c7\x2dinit-merged.mount: Succeeded.
Jun 23 09:06:08 gandalf systemd[1]: var-lib-docker-overlay2-158dbf596177fea1dbaf78a255fea9366e73a17c99031f313203847b47d264c7\x2dinit-merged.mount: Succeeded.
Jun 23 09:06:08 gandalf NetworkManager[964]: <info> [1624453568.0145] manager: (veth4f08fa5): new Veth device (/org/freedesktop/NetworkManager/Devices/9)
Jun 23 09:06:08 gandalf NetworkManager[964]: <info> [1624453568.0155] manager: (veth0426df2): new Veth device (/org/freedesktop/NetworkManager/Devices/10)
Jun 23 09:06:08 gandalf systemd-udevd[3494]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 23 09:06:08 gandalf systemd-udevd[3494]: Using default interface naming scheme 'v247'.
Jun 23 09:06:08 gandalf systemd-udevd[3493]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 23 09:06:08 gandalf systemd-udevd[3493]: Using default interface naming scheme 'v247'.
Jun 23 09:06:08 gandalf networkd-dispatcher[972]: WARNING:Unknown index 7 seen, reloading interface list
Jun 23 09:06:08 gandalf gnome-shell[1738]: Removing a network device that was not added
Jun 23 09:06:08 gandalf gnome-shell[1738]: Removing a network device that was not added
Jun 23 09:06:08 gandalf systemd[1]: Unnecessary job for /sys/subsystem/net/devices/veth0426df2 was removed.
Jun 23 09:06:08 gandalf systemd[1]: Unnecessary job for /sys/subsystem/net/devices/veth4f08fa5 was removed.
Jun 23 09:06:08 gandalf systemd[1]: Started Netscript ifup for veth0426df2.
Jun 23 09:06:08 gandalf systemd[1]: Started Netscript ifup for veth4f08fa5.
Jun 23 09:06:08 gandalf systemd[1]: Stopping Netscript ifup for veth0426df2...
Jun 23 09:06:08 gandalf systemd[1]: Stopping Netscript ifup for veth4f08fa5...
Jun 23 09:06:08 gandalf dockerd[2556]: time="2021-06-23T09:06:08.061241688-04:00" level=warning msg="Failed to delete host side interface (veth0426df2)'s link" error="no such device"
Jun 23 09:06:08 gandalf systemd[1548]: var-lib-docker-overlay2-158dbf596177fea1dbaf78a255fea9366e73a17c99031f313203847b47d264c7-merged.mount: Succeeded.
Jun 23 09:06:08 gandalf systemd[1386]: var-lib-docker-overlay2-158dbf596177fea1dbaf78a255fea9366e73a17c99031f313203847b47d264c7-merged.mount: Succeeded.
Jun 23 09:06:08 gandalf systemd[1]: var-lib-docker-overlay2-158dbf596177fea1dbaf78a255fea9366e73a17c99031f313203847b47d264c7-merged.mount: Succeeded.
Jun 23 09:06:08 gandalf dockerd[2556]: time="2021-06-23T09:06:08.072668630-04:00" level=error msg="0c5f9a496d902f584970a1f58a6babe9c2e0beb405922119512693ddd4db02df cleanup: failed to delete container from containerd: no such container"
Jun 23 09:06:08 gandalf dockerd[2556]: time="2021-06-23T09:06:08.076781116-04:00" level=error msg="Handler for POST /v1.41/containers/0c5f9a496d902f584970a1f58a6babe9c2e0beb405922119512693ddd4db02df/start returned error: failed to create endpoint amazing_noether on network bridge: adding interface veth0426df2 to bridge docker0 failed: could not find bridge docker0: route ip+net: no such network interface"
Jun 23 09:06:08 gandalf sh[3525]: Usage: netscript ifup|ifdown|ifqos|ifreload
Jun 23 09:06:08 gandalf sh[3525]: {eth0|enp38s0|wlp37s0|all}
Jun 23 09:06:08 gandalf sh[3521]: Usage: netscript ifup|ifdown|ifqos|ifreload
Jun 23 09:06:08 gandalf sh[3521]: {eth0|enp38s0|wlp37s0|all}
Jun 23 09:06:08 gandalf systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE
Jun 23 09:06:08 gandalf systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE
Jun 23 09:06:08 gandalf netscript[3558]: Usage: netscript ifup|ifdown|ifqos|ifreload
Jun 23 09:06:08 gandalf netscript[3558]: {eth0|enp38s0|wlp37s0|all}
Jun 23 09:06:08 gandalf netscript[3563]: Usage: netscript ifup|ifdown|ifqos|ifreload
Jun 23 09:06:08 gandalf netscript[3563]: {eth0|enp38s0|wlp37s0|all}
Jun 23 09:06:08 gandalf systemd[1]: [email protected]: Control process exited, code=exited, status=1/FAILURE
Jun 23 09:06:08 gandalf systemd[1]: [email protected]: Failed with result 'exit-code'.
Jun 23 09:06:08 gandalf systemd[1]: Stopped Netscript ifup for veth0426df2.
Jun 23 09:06:08 gandalf systemd[1]: [email protected]: Control process exited, code=exited, status=1/FAILURE
Jun 23 09:06:08 gandalf systemd[1]: [email protected]: Failed with result 'exit-code'.
Jun 23 09:06:08 gandalf systemd[1]: Stopped Netscript ifup for veth4f08fa5.
Last interesting note: when I first start the docker daemon I can indeed see the docker0 interface in ifconfig. But after shutting down the first successful running container the docker0 interface disappear.
Not sure if it will help, had a similar issue and found that it was related to the following package 'netscript-2.4' I installed it to make dealing with network from CLI easier, then the issues started. removed and they went away.
Couldn't get to the bottom of why but.
Anyway, Removing the package and issue went away.
"sudo apt remove netscript-2.4"
good luck.
Not sure if it will help, had a similar issue and found that it was related to the following package 'netscript-2.4' I installed it to make dealing with network from CLI easier, then the issues started. removed and they went away.
Couldn't get to the bottom of why but.
Anyway, Removing the package and issue went away. "sudo apt remove netscript-2.4"
good luck.
Solved the problem for me, you saved my day
Not sure if it will help, had a similar issue and found that it was related to the following package 'netscript-2.4' I installed it to make dealing with network from CLI easier, then the issues started. removed and they went away. Couldn't get to the bottom of why but. Anyway, Removing the package and issue went away. "sudo apt remove netscript-2.4" good luck.
Solved the problem for me, you saved my day
Solved for me too, thanks
Not sure if it will help, had a similar issue and found that it was related to the following package 'netscript-2.4' I installed it to make dealing with network from CLI easier, then the issues started. removed and they went away.
Couldn't get to the bottom of why but.
Anyway, Removing the package and issue went away. "sudo apt remove netscript-2.4"
good luck.
@l3Gopher
Yoo bro, thank you so much, I lost 5h today, you saved my life 😇
@l3Gopher wow, this git had nothing to do with me but I had the same error, wasted about 5 hrs too. This fixed it thanks!
Interesting, so yes, this looks to be another service (netscript) that attempts to manage interfaces, and from the comments above, it looks like it kicks the interface from under the docker daemon.
So, it doesn't appear to be a bug in this repository; the only "solution" I could come up with would be the docker service to keep monitoring the interface and recreate it if it gets removed, but likely that won't help (it would only result in two services fighting ownership of the interface).
Suggestions I have;
- Perhaps
netscripthas a configuration option to exclude certain interfaces; I had a quick look at the netscript manpage; for current versions of Ubuntu, that page points toifup, but it's very minimal, and their manpage for the configuration file is only a placeholder ipfilter.conf; if someone knows how to configure it to exclude, that could be useful information to add to the Docker documentation. - Perhaps a mention of this issue could be added to the troubleshooting section in the docs; contributions to the documentation are welcome; let me know if anyone is interested.
Not sure if it will help, had a similar issue and found that it was related to the following package 'netscript-2.4' I installed it to make dealing with network from CLI easier, then the issues started. removed and they went away.
Couldn't get to the bottom of why but.
Anyway, Removing the package and issue went away. "sudo apt remove netscript-2.4"
good luck.
Awesome! Solved the problem for me, thanks! 😁
Not sure if it will help, had a similar issue and found that it was related to the following package 'netscript-2.4' I installed it to make dealing with network from CLI easier, then the issues started. removed and they went away.
Couldn't get to the bottom of why but.
Anyway, Removing the package and issue went away. "sudo apt remove netscript-2.4"
good luck.
Solved the problem for me, thanks a lot!
Interesting, so yes, this looks to be another service (netscript) that attempts to manage interfaces, and from the comments above, it looks like it kicks the interface from under the docker daemon.
So, it doesn't appear to be a bug in this repository; the only "solution" I could come up with would be the docker service to keep monitoring the interface and recreate it if it gets removed, but likely that won't help (it would only result in two services fighting ownership of the interface).
Suggestions I have;
* Perhaps `netscript` has a configuration option to exclude certain interfaces; I had a quick look at the [netscript manpage](https://manpages.ubuntu.com/manpages/xenial/man8/netscript-ipfilter.8.html); for current versions of Ubuntu, that page points to [`ifup`](https://manpages.ubuntu.com/manpages/jammy/man8/netscript-ipfilter.8.html), but it's very minimal, and their manpage for the configuration file is only a placeholder [ipfilter.conf](https://manpages.ubuntu.com/manpages/xenial/man5/ipfilter.conf.5.html); if someone knows how to configure it to exclude, that could be useful information to add to the Docker documentation. * Perhaps a mention of this issue could be added to the [troubleshooting section in the docs](https://docs.docker.com/engine/install/troubleshoot/); contributions to the documentation are welcome; let me know if anyone is interested.
@thaJeztah I had the same issue on Debian 10 and prevented netscript from messing up by creating /etc/network/interfaces.d/20-docker0 with this content:
iface docker0 inet manual
Not sure if it will help, had a similar issue and found that it was related to the following package 'netscript-2.4' I installed it to make dealing with network from CLI easier, then the issues started. removed and they went away.
Couldn't get to the bottom of why but.
Anyway, Removing the package and issue went away. "sudo apt remove netscript-2.4"
good luck.
It worked !!! Thank You It took a long, but only this reference available over internet worked Note- Please, Don't go for kernel upgrade / downgrade .. try it once then go for other solutions
Not sure if it will help, had a similar issue and found that it was related to the following package 'netscript-2.4' I installed it to make dealing with network from CLI easier, then the issues started. removed and they went away.
Couldn't get to the bottom of why but.
Anyway, Removing the package and issue went away. "sudo apt remove netscript-2.4"
good luck.
Thanks Alot Brother
Not sure if it will help, had a similar issue and found that it was related to the following package 'netscript-2.4' I installed it to make dealing with network from CLI easier, then the issues started. removed and they went away.
Couldn't get to the bottom of why but.
Anyway, Removing the package and issue went away. "sudo apt remove netscript-2.4"
good luck.
Thank you so much this is the Solution.
Not sure if it will help, had a similar issue and found that it was related to the following package 'netscript-2.4' I installed it to make dealing with network from CLI easier, then the issues started. removed and they went away.
Couldn't get to the bottom of why but.
Anyway, Removing the package and issue went away. "sudo apt remove netscript-2.4"
good luck.
You are a hero. Thx!
@dvdksn There seems to be a few people impacted by netscript, netplan and even networkd messing with Docker interfaces (see https://github.com/moby/moby/issues/40217). Would you mind updating the Troubleshoot doc page to include something about them?
Regarding, netscript we can recommend to either:
- Use a config file to tell it not to mess with
docker0(see this comment). But this wouldn't work with custom networks since their interfaces have non-predictable names ; - Or to uninstall / disable it altogether.
For netplan and networkd, a few fixes proposed in the linked issue (see https://github.com/moby/moby/issues/40217#issuecomment-2041628162 and https://github.com/moby/moby/issues/40217#issuecomment-1551515509). Tell me if you need help / confirmation of what's needed for them.
