compose icon indicating copy to clipboard operation
compose copied to clipboard
verified publisher icon docker

Unable to connect to custom bridge networks with manual subnet, "route for the gateway could not be found."

Open njalooo opened this issue 3 years ago • 6 comments

BUG REPORT INFORMATION

Unable to connect to custom created bridge networks, which have a manual subnet. No matter if created from the docker-compose.yml file or manually.

Steps to reproduce the issue:

  1. docker network create --subnet 127.29.0.0/16 --gateway 127.29.0.1 --attachable wgtraefik
  2. docker-compose.yml:
---
version: "2.1"
services:
  wireguard:
    image: lscr.io/linuxserver/wireguard:latest

    container_name: wireguard

    cap_add:
      - NET_ADMIN
      - SYS_MODULE

    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Zurich

    volumes:
      - ./wireguard:/config
      - /lib/modules:/lib/modules:ro

    sysctls:
      - net.ipv4.conf.all.src_valid_mark=1
      - net.ipv6.conf.all.disable_ipv6=1
    restart: unless-stopped

    cap_add:
      - NET_ADMIN
      - SYS_MODULE

    networks:
      wgtraefik:
        ipv4_address: 127.29.8.7

networks:
  wgtraefik:
    name: wgtraefik
    external: true
  1. docker-compose up

Describe the results you received:

[+] Running 7/7
 ⠿ wireguard Pulled                                                                                                                  76.4s
   ⠿ ed555500656f Pull complete                                                                                                      24.2s
   ⠿ 1698c149817b Pull complete                                                                                                      24.8s
   ⠿ ef9834f45802 Pull complete                                                                                                      26.8s
   ⠿ 6171437a7e1a Pull complete                                                                                                      26.9s
   ⠿ 01dde983cc72 Pull complete                                                                                                      68.0s
   ⠿ cc06d3be5aa4 Pull complete                                                                                                      68.1s
[+] Running 0/1
 ⠿ Container wireguard  Starting                                                                                                      0.7s
Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2022-09-12T15:57:21+02:00" level=fatal msg="failed to set gateway while updating gateway: route for the gateway 127.29.0.1 could not be found: invalid argument": unknown

Describe the results you expected: Container starting and connecting to network with custom ip.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker compose version:

Docker Compose version 2.9.0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
  compose: Docker Compose (Docker Inc., 2.9.0)

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 1
 Server Version: 20.10.17
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6.m
 runc version: 
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.19.1-3-MANJARO
 Operating System: Manjaro Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 24
 Total Memory: 94.29GiB
 Name: Macpro-Linux-Server
 ID: 6XZG:L2PM:H3A6:BG7Q:XITI:K2KK:TNEL:ZVIL:LEEV:PLV5:THN2:JSTE
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details: Relevant ip addr entry:

br-7ec8e3ab540e: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:2c:e7:01:c7 brd ff:ff:ff:ff:ff:ff
    inet 127.29.0.1/16 brd 127.29.255.255 scope global br-7ec8e3ab540e
       valid_lft forever preferred_lft forever

njalooo avatar Sep 12 '22 14:09 njalooo

I tried reinstalling docker and docker-compose by the way, including deleting the /var/lib/docker/ folder.

njalooo avatar Sep 12 '22 14:09 njalooo

Hi @njalooo, thanks for the report. Can you provide some additional information, such as if this bug is new (did this work in previous compose versions?) and whether this works outside of compose (could you provide a working docker run command for running the wireguard in an analogous usecase)?

laurazard avatar Sep 12 '22 14:09 laurazard

Hey, this is a fresh install, so i don't know about prior versions. Attaching to networks without an specified subnetmask works fine, so i didn't notice for a while.

As to answer you second Question: docker pull hello-world docker network create --subnet 127.29.0.0/16 --gateway 127.29.0.1 --attachable hello-worldtest docker run -d --network hello-worldtest --ip 127.29.1.1 hello-world Returns the same Error: docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2022-09-12T21:32:00+02:00" level=fatal msg="failed to set gateway while updating gateway: route for the gateway 127.29.0.1 could not be found: invalid argument": unknown.

I installed Linux Manjaro on an 2008(?) Mac Pro, and updated to the latest generic Linux kernel, and updated all packages. Everything else seems to work just fine.

njalooo avatar Sep 12 '22 19:09 njalooo

This is the entire console history since install, in case you might spot an issue with the setup:

sudo pacman -Syu
sudo systemctl start sshd.service
sudo pacman -S linux519

//I had some issues with connecting to my wifi over the GUI

lspci
lspci -vnn -d 14e4:
rmmod --help
sudo pacman -S git
cd /opt
sudo git clone https://aur.archlinux.org/yay-git.git
sudo chown -R caramon:caramon ./yay-git
cd yay-git
sudo pacman -Ss linux519
sudo pacman -S linux519-headers
make -version
qmake -version
sudo pacman -S base-devel
makepkg -si
yay -S
yay -u
yay -Syu
yay -S b43-firmware
sudo pacman -S manjaro-firmware
yay -S b43-firmware-classic
cd
sudo pacman -S docker cocker-compose
lsmod
lsmod | grep b43
modprobe -r b43
sudo modprobe -r b43
sudo modprobe b43
_iwconfig
ifconfig
ip link set wlan0 up
sudo ip link set wlan0 up
ip list wlan0 scn
lspci -k
sudo nmtui

//Issue solved over command line util

sudo systemctl enable docker.service
sudo systemctl enable docker.socket
sudo systemctl start docker.socket
sudo docker run hello-world
sudo docker sp
sudo docker rmi hello-world
sudo docker rmi -f hello-world
sudo docker container rm cb40d4ec38ff
sudo docker-compose --version
docker volume create portainer_data
whoami
sudo docker volume create portainer_data
docker run -d -8000:8000 -p 9000:9000 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce
docker run -d -p 8000:8000 -p 9000:9000 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce
sudo -s
sudo systemctl enable sshd.service
iptables list
iptables -l
iptables -h
iptables -L
iptables -list
sudo iptables -list
sudo systemctl 
sudo systemctl --help
sudo systemctl status sshd
ssh [email protected]
docker-compose stop
sudo docker container list
sudo docker ps
sudo docker-compose ps
sudo docker network rm none
sudo docker network rm host
docker-compose network ls
sudo docker network rm wireguard
sudo docker images
sudo docker network ls
sudo docker network create wireguard
sudo docker-compose down
sudo chmod -R o-rwx wireguard
sudo chmod o+rx wireguard
cat wg0.conf
sudo docker-compose up
nano wg0.conf
sudo docker-compose exec wireguard /bin/bash
ip adr
cd /docker/portainer
sudo nano /etc/sysctl.d
sudo rm -rf wireguard
sudo mkdir wireguard
sudo touch wg0
sudo chmod 
sudo chmod go -rwx
sudo chmod -R go-rwx wireguard
cd -s wireguard
sudo cd wireguard
sudo chmod go+rx
sudo chmod go+rx wireguard
cd wireguard
cat wg0
sudo mv wg0 wg0.conf
sudo nano docker-compose.yml
sudo nano /etc/passwd
nano docker-compose.yml
pacman -S vim
sudo pacman -S vim
vim docker-compose.yml
touch ./traefik/traefik.yml
sudo touch ./traefik/traefik.yml
sudo vim ./traefik/traefik.yml
iptables --help
sudo cat ./wireguard/wg0.conf
man iptables
ls traefik
sudo rm -rf
sudo rm -rf ./traefik
sudo mkdir traefik
sudo touch acme.json
sudo chmod 600 acme.json
sudo touch traefik.yml
vim traefik.yml
docker network create proxy
docker network rm proxy
sudo touch docker-compose.yml
touch config.yml
sudo touch config.yml
sudo vim config.yml
cd 
sudo pacman -S apache2-utils
cd /docker/wireguard/wireguard
sudo vim wg0.conf
docker-compose dow
sudo vim traefik

//I rebooted the server later on, so the following IPtables shouldn't affect anything

iptables
iptables --list
sudo iptables -A INPUT -s 192.168.178.0/24 -m conntrack --cstate NEW,ESTABLISHED,RELATED -j ACCEPT
sudo iptables -A INPUT -s 192.168.178.0/24 -m conntrack --ctstate NEW,ESTABLISHED,RELATED -j ACCEPT
sudo iptables -A OUTPUT -d 192.168.178.0/24 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
sudo iptables -A OUTPUT -d 192.168.178.0/24 -m conntrack --ctstate NEW,INVALID -j REJECT
sudo iptables -P INPUT DROP
sudo iptables -D INPUT -s 192.168.178.0/24 -m conntrack --ctstate NEW,ESTABLISHED,RELATED -j ACCEPT
sudo iptables -A INPUT -s 192.168.178.0/24 -J ACCEPT
sudo iptables -A INPUT -s 192.168.178.0/24 -j ACCEPT
sudo iptables -I DOCKER-USER -i enp9s0,enp10s0,wlan0 -conntrack --ctstate ESTABLISHED,RELATED -ACCEPT
sudo iptables -I DOCKER-USER -i enp+s0 -conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
sudo iptables -I DOCKER-USER -i enp+s0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
sudo iptables -I DOCKER-USER -i wlan0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
sudo iptables -I DOCKER-USER -i enp+s0 -m conntrack --ctstate NEW,INVALID -j DROP
sudo iptables -I DOCKER-USER -i wlan0 -m conntrack --ctstate NEW,INVALID -j DROP
sudo iptables -I DOCKER-USER -i enp+s0 -s 192.168.178.0/24 -m conntrack --ctstate NEW,ESTABLISHED,RELATED -j ACCEPT
sudo iptables -I DOCKER-USER -i wlan0 -s 192.168.178.0/24 -m conntrack --ctstate NEW,ESTABLISHED,RELATED -j ACCEPT
sudo iptables -I DOCKER-USER -o enp+s0 -m conntrack --ctstate NEW,ESTABLISHED,RELATED -j ACCEPT
sudo iptables -I DOCKER-USER -o wlan0 -m conntrack --ctstate NEW,ESTABLISHED,RELATED -j ACCEPT
sudo iptables -I DOCKER-USER -o enp+s0 -d 192.168.0.0/16 -m conntrack --ctstate NEW,INVALID -j DROP
sudo iptables -D DOCKER-USER -o enp+s0 -d 192.168.0.0/16 -m conntrack --ctstate NEW,INVALID -j DROP
sudo iptables -I DOCKER-USER -o enp+s0 -d 192.168.0.0/16 -m conntrack --ctstate NEW,INVALID -j REJECT
sudo iptables -I DOCKER-USER -o wlan0 -d 192.168.0.0/16 -m conntrack --ctstate NEW,INVALID -j REJECT
sudo iptables -I DOCKER-USER -o enp+s0 -d 192.168.178.0/24 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
sudo iptables -I DOCKER-USER -o wlan0 -d 192.168.178.0/24 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
histo
history
sudo iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
history -list
history --list
cat $HISTORYFILE
cat $HISTFILE
cd traefik
ls -al
cd ../traefik
docker network --help
docker network create --subnet 127.29.0.0/16 wgtraefik
sudo iptables
sudo iptables --list
cd ../wireguard
docker network create --subnet 127.29.0.0/16 --gateway 127.29.0.1 wgtraefik
docker network create --help
docker network create --subnet 127.29.0.0/16 --gateway 127.29.0.1 --attachable -d bridge wgtraefik
cat /etc/docker/
ls /etc/docker/
cat /etc/docker/key.json
sudo cat /etc/docker/key.json
docker network create --subnet 127.29.0.0/16 --attachable wgtraefik
sudo pacman -R docker-compose
sudo pacman --help
sudo pacman -R docker
docker network create --subnet 127.29.0.0/16 --gateway 127.29.0.1/32 --attachable wgtraefik
docker network create --subnet 127.29.0.0/16 --gateway 127.29.0.1/16 --attachable wgtraefik
docker ps
sudo pacman -R docker docker-compose
ös
cd etc
cd docker
cd var/lib
sudo rm -rf docker
cd /
reboot
sudo pacman -S docker docker-compose
sudo systemctl status docker.service
sudo systemctl status docker.socket
usermod --help
sudo usermod -aG docker caramon
sudo docker network create --subnet 127.29.0.0/16 --gateway 127.29.0.1 --attachable wgtraefik
sudo systemctl start docker.socket 
sudo systemctl start docker.service
ip route show
cat docker-compose.yml
docker-compose --version
docker info
uname
uname -a
docker network list
docker network create --subnet 127.29.0.0/16 --gateway 127.29.0.1 --attachable wgtraefik
ip addr
docker network rm wgtraefik
docker network create wgtraefik
cd /docker/wireguard
docker-compose exec wireguard /bin/bash
docker network inspect wgtraefik
docker network create traefik
cd /docker/traefik
ls
cd ./traefik
docker-compose up -d
sudo vim docker-compose.yml
cd ..
docker-compose exec traefik /bin/bash
docker-compose exec traefik /bin/sh
cd /docker/traefik/traefik
sudo vim traefik.yml
docker-compose down
docker-compose up
exit
cd ..
cat docker-compose.yml
cd traefik
cat traefik.yml
docker-compose down
sudo touch config.yml
ls
docker-compose up
sudo touch config.yml
docker-compose down
sudo vim docker-compose.yml
docker-compose up
docker-compose down
sudo vim docker-compose.yml
docker-compose up -d
docker ps
ls
docker pull hello-world
docker run --help
docker network list
docker network create --subnet 127.29.0.0/16 --gateway 127.29.0.1 --attachable hello-worldtest
docker network inspect hello-worldtest
docker run -d --network hello-worldtest --ip 127.29.1.1 hello-world

njalooo avatar Sep 12 '22 19:09 njalooo

And the Output of lspci, in case it might be a hardware/driver related issue.


00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 22)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 22)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 22)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 22)
00:0d.0 Host bridge: Intel Corporation Device 343a (rev 22)
00:0d.1 Host bridge: Intel Corporation Device 343b (rev 22)
00:0d.2 Host bridge: Intel Corporation Device 343c (rev 22)
00:0d.3 Host bridge: Intel Corporation Device 343d (rev 22)
00:0d.4 Host bridge: Intel Corporation 7500/5520/5500/X58 Physical Layer Port 0 (rev 22)
00:0d.5 Host bridge: Intel Corporation 7500/5520/5500 Physical Layer Port 1 (rev 22)
00:0d.6 Host bridge: Intel Corporation Device 341a (rev 22)
00:0d.7 Host bridge: Intel Corporation Device 341b (rev 22)
00:0e.0 Host bridge: Intel Corporation Device 341c (rev 22)
00:0e.1 Host bridge: Intel Corporation Device 341d (rev 22)
00:0e.2 Host bridge: Intel Corporation Device 341e (rev 22)
00:0e.3 Host bridge: Intel Corporation Device 341f (rev 22)
00:0e.4 Host bridge: Intel Corporation Device 3439 (rev 22)
00:0f.0 Performance counters: Intel Corporation Device 3424 (rev 22)
00:10.0 PIC: Intel Corporation 7500/5520/5500/X58 Physical and Link Layer Registers Port 0 (rev 22)
00:10.1 PIC: Intel Corporation 7500/5520/5500/X58 Routing and Protocol Layer Registers Port 0 (rev 22)
00:11.0 PIC: Intel Corporation 7500/5520/5500 Physical and Link Layer Registers Port 1 (rev 22)
00:11.1 PIC: Intel Corporation 7500/5520/5500 Routing & Protocol Layer Register Port 1 (rev 22)
00:13.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller (rev 22)
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 22)
00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 22)
00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 22)
00:14.3 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Throttle Registers (rev 22)
00:15.0 PIC: Intel Corporation 7500/5520/5500/X58 Trusted Execution Technology Registers (rev 22)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1b.0 Audio device: Intel Corporation 82801JI (ICH10 Family) HD Audio Controller
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1
00:1c.1 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 2
00:1c.2 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 3
00:1c.3 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 4
00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5
00:1c.5 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIB (ICH10) LPC Interface Controller
00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
01:00.0 PCI bridge: Microsemi / PMC / IDT PES12T3G2 PCI Express Gen2 Switch (rev 01)
02:02.0 PCI bridge: Microsemi / PMC / IDT PES12T3G2 PCI Express Gen2 Switch (rev 01)
02:04.0 PCI bridge: Microsemi / PMC / IDT PES12T3G2 PCI Express Gen2 Switch (rev 01)
04:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04)
05:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 5770]
05:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series]
06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 5770]
06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series]
09:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
0a:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
0b:00.0 PCI bridge: Texas Instruments XIO2213A/B/XIO2221 PCI Express to PCI Bridge [Cheetah Express] (rev 01)
0c:00.0 FireWire (IEEE 1394): Texas Instruments XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express] (rev 01)
0d:00.0 Network controller: Broadcom Inc. and subsidiaries BCM4322 802.11a/b/g/n Wireless LAN Controller (rev 01)

njalooo avatar Sep 12 '22 19:09 njalooo

@thaJeztah this looks like a https://github.com/moby/moby (or maybe runc) issue to me, wdyt?

laurazard avatar Sep 12 '22 22:09 laurazard

Closing, please open an issue over in https://github.com/moby/moby if you like.

laurazard avatar Jan 24 '23 12:01 laurazard