moby
Failed to set up container networking 28.0.1 (module IP_NF_RAW dependency)
Description
Docker networking appears to still be broken in 28.0.1.
I am running a fresh install of the NVIDIA Jetson Linux on an NVIDIA Jetson Orin Nano.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.5 LTS
Release: 22.04
Codename: jammy
Docker version 28.0.1, build 068a01e
I have made no other changes than to copy the docker-compose.yaml file to the machine.
dave@ubuntu:~/ollama$ docker compose up
[+] Running 21/21
✔ ollama Pulled 104.0s
✔ 1b9f3c55f9d4 Pull complete 2.8s
✔ b48598350c3a Pull complete 3.1s
✔ c675bebda2db Pull complete 3.5s
✔ 42006779754c Pull complete 103.3s
✔ openWebUI Pulled 87.2s
✔ d51c377d94da Pull complete 3.6s
✔ 987cac002684 Pull complete 3.9s
✔ 076b75118273 Pull complete 5.4s
✔ 157e623d2984 Pull complete 5.4s
✔ 40d5353a5918 Pull complete 5.4s
✔ 4f4fb700ef54 Pull complete 5.4s
✔ aebeb0b4e5d0 Pull complete 5.4s
✔ 03f562834d64 Pull complete 5.4s
✔ dc0f62a912f5 Pull complete 32.4s
✔ d5719fd73d52 Pull complete 32.4s
✔ d32514fe2679 Pull complete 80.9s
✔ 76d16054d9ea Pull complete 85.5s
✔ 20c2e04f7bfa Pull complete 85.5s
✔ 5bb61eb10273 Pull complete 85.6s
✔ 445ddbefd034 Pull complete 86.5s
[+] Running 3/3
✔ Network ollama_default Created 0.1s
✔ Container ollama Created 0.1s
✔ Container open-webui Created 0.0s
Attaching to ollama, open-webui
Gracefully stopping... (press Ctrl+C again to force)
Error response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint ollama (28422f573e9a997429ed96a5f6d731e223c725cbe8d35d2c9dfac8f966a73adc): Unable to enable DIRECT ACCESS FILTERING - DROP rule: (iptables failed: iptables --wait -t raw -A PREROUTING -p tcp -d 172.18.0.2 --dport 11434 ! -i br-2d1c35b32c76 -j DROP: iptables v1.8.7 (legacy): can't initialize iptables table `raw': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
(exit status 3))
Here is the pretty basic docker-compose.yaml: services:
openWebUI:
container_name: open-webui
image: ghcr.io/open-webui/open-webui:main
restart: unless-stopped
ports:
- "8080:8080"
volumes:
- /open-webui:/app/backend/data
depends_on:
- ollama
environment:
- WEBUI_AUTH=False
- OLLAMA_BASE_URL=http://localhost:11434
ollama:
container_name: ollama
image: ollama/ollama:latest
# image: dustynv/ollama:main-r36.4.0
runtime: nvidia
pull_policy: always
restart: unless-stopped
ports:
- "11434:11434"
volumes:
- /ollama:/root/.ollama
environment:
- OLLAMA_KEEP_ALIVE=24h
- OLLAMA_HOST=0.0.0.0:11434
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
Reproduce
docker compose up
Expected behavior
The containers should come up and run properly.
If I downgrade to 27.x everything works as expected.
sudo apt install docker-ce=5:27.*
docker version
dave@ubuntu:~/ollama$ docker version
Client: Docker Engine - Community
Version: 28.0.1
API version: 1.47 (downgraded from 1.48)
Go version: go1.23.6
Git commit: 068a01e
Built: Wed Feb 26 10:41:16 2025
OS/Arch: linux/arm64
Context: default
Server: Docker Engine - Community
Engine:
Version: 27.5.1
API version: 1.47 (minimum version 1.24)
Go version: go1.22.11
Git commit: 4c9b3b0
Built: Wed Jan 22 13:41:23 2025
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.7.25
GitCommit: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
runc:
Version: 1.2.4
GitCommit: v1.2.4-0-g6c52b3f
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
dave@ubuntu:~/ollama$ docker info
Client: Docker Engine - Community
Version: 28.0.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.21.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.33.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 2
Running: 0
Paused: 0
Stopped: 2
Images: 2
Server Version: 27.5.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 nvidia runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
runc version: v1.2.4-0-g6c52b3f
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.148-tegra
Operating System: Ubuntu 22.04.5 LTS
OSType: linux
Architecture: aarch64
CPUs: 6
Total Memory: 7.441GiB
Name: ubuntu
ID: ccb022f3-ed3d-49c0-a849-dc12b1483aee
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional Info
No response
@virtualJonesie Thanks for reporting.
Could you paste here the output of zgrep IP_NF_RAW /proc/config.gz please?
# CONFIG_IP_NF_RAW is not set
This is an out of the box install. After quite a bit of troubleshooting yesterday, I decided to start fresh this morning.
Yikes! I'm not familiar with Nvidia Jetson Linux -- do you have linux-modules-<something-something> available in your repos? I'd try to install it to make the iptable_raw module available.
Yikes! I'm not familiar with Nvidia Jetson Linux -- do you have
linux-modules-<something-something>available in your repos? I'd try to install it to make theiptable_rawmodule available.
It's basically Ubuntu 22.04.
Let me see what I can find to answer your question about the modules.
Same here. Not working
27.5.1 - fine 28.0 - doesn’t work 28.0.1 - doesn’t work
Raspi pi 5. RaspiOS latest. Testing with qbitorrent container funnelled through PIA wireguard container
Rolled back to 27.5.1, all smiles.
Edit, I should mention, RaspiOS lite with OMV 7.7. Rock solid. But could be something there?
Private Internet Access Wireguard VPN is set up as a service called “vpn” in my portainer stack with all necessary ports open for its children.
Qbittorrent, and a bunch of other containers, talk to it using network mode: “service:vpn” to funnel through it, which has been working great for two years or so :)
I’m not dead savvy on all the terms and minutia bit that should illustrate the flow
Same here. Not working
27.5.1 - fine 28.0 - doesn’t work 28.0.1 - doesn’t work
Raspi pi 5. RaspiOS latest. Testing with qbitorrent container funnelled through PIA wireguard container
Rolled back to 27.5.1, all smiles.
I am experiencing the same version-success. Anything 28.x fails.
Flushed iptables, upgraded to 28.0.1, reboot. No connection
Downgraded back to 27.5.1. Perfect.
Have it on apt-mark hold until someone considerably smarter than me has any insights ;)
I'm having the same problem on my machine, NVIDIA Jetson Orin Nano Super dev. kit.
OS is Jetson Linux r36.4.3 (Ubuntu 22.04.5)
$ uname -srvp
Linux 5.15.148-tegra #1 SMP PREEMPT Tue Jan 7 17:14:38 PST 2025 aarch64
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
$ cat /etc/nv_tegra_release
# R36 (release), REVISION: 4.3, GCID: 38968081, BOARD: generic, EABI: aarch64, DATE: Wed Jan 8 01:49:37 UTC 2025
# KERNEL_VARIANT: oot
TARGET_USERSPACE_LIB_DIR=nvidia
TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia
$ zgrep IP_NF_RAW /proc/config.gz
# CONFIG_IP_NF_RAW is not set
No problem on Docker version 27.5.1
$ docker --version
Docker version 27.5.1, build 9f9e405
$ docker run -dit -p 80:80 --rm --name alpine alpine:latest
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
e980327524da alpine:latest "/bin/sh" 46 seconds ago Up 45 seconds 0.0.0.0:80->80/tcp, :::80->80/tcp alpine
Cannot publish port on Docker version 28.0.1
$ docker --version
Docker version 28.0.1, build 068a01e
$ docker run -dit -p 80:80 --rm --name alpine alpine:latest
44de6abaa53736b9ffa4eb5171b631475772b015aa9dc8f92005065c016685d8
docker: Error response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint alpine (d373025c7fbf1f880fe55171c6673b2b8f036ee84160128712ffde9980c02266): Unable to enable DIRECT ACCESS FILTERING - DROP rule: (iptables failed: iptables --wait -t raw -A PREROUTING -p tcp -d 172.17.0.2 --dport 80 ! -i docker0 -j DROP: iptables v1.8.7 (legacy): can't initialize iptables table `raw': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
(exit status 3))
Run 'docker run --help' for more information
publishing no port on Docker version 28.0.1
$ docker run -dit --rm --name alpine alpine:latest
8f7ad861dbd344e7071dbd160d76b8156b6da181420a0e9df98e7d4733db3947
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8f7ad861dbd3 alpine:latest "/bin/sh" 34 seconds ago Up 33 seconds alpine
$ docker exec -it alpine /bin/sh
/ # uname -a
Linux 8f7ad861dbd3 5.15.148-tegra #1 SMP PREEMPT Tue Jan 7 17:14:38 PST 2025 aarch64 Linux
/ # exit
Hi all - I think this is only an issue when iptables --version is iptables-legacy, iptables-nft doesn't seem to need the IP_NF_RAW module.
But, I see in a couple of Nvidia forum posts people have had issues with docker and iptables-nft, and reconfiguring to use iptables-legacy helped ... is that the case here?
If not, switching to iptables-nft might be the easiest workaround ...
sudo update-alternatives --set iptables /usr/sbin/iptables-nft
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-nft
(Also switch arptables and ebtables if those are configured to use legacy. To switch back, swap *-nft for *-legacy in the commands.)
I'd be interested to hear if that works ... or, if there's also an issue with iptables-nft, I'd like to understand that too. (I think most OSs now use it my default, and we use it in moby regression testing. So it should be ok, but perhaps there's some specific issue here.)
Hey Rob. Thanks a mil for your hard work here
RaspiOS 64 lite. My iptables version is:
iptables v1.8.9 (nf_tables)
So I think this is iptables nft or?
Thanks @braymullo - yes, that's the one I thought would work! I tried it on a Debian kernel built without that module, but there must be some other difference.
Just to check, with moby 28.0.1, you're getting the error message that includes "can't initialize iptables table `raw'"?
Hey Rob,
I’d be happy to check for you tomorrow on my home lab. I use portainer, so that specific error message o don’t think would be clear to me. Is there a way for me to check, or an article you could link me to to brush up my knowledge on debugging docker so I can better help?
I tried with my Jetson Linux(Ubuntu22.04) and Raspberry Pi 4B (Ubuntu 20.04)
First, Jetson Linux(Ubuntu22.04)
## Jetson Linux has no iptable_raw.ko file.
$ find /lib/modules/$(uname -r) -name 'iptable_raw.ko'
$ lsmod | grep -i iptable_raw
## iptables is legacy
$ iptables --version
iptables v1.8.7 (legacy)
## arptables and ebtables are nf_tables
$ arptables --version
arptables v1.8.7 (nf_tables)
$ ebtables --version
ebtables 1.8.7 (nf_tables)
## switch to iptables-nft
$ sudo update-alternatives --set iptables /usr/sbin/iptables-nft
update-alternatives: using /usr/sbin/iptables-nft to provide /usr/sbin/iptables (iptables) in manual mode
$ sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-nft
update-alternatives: using /usr/sbin/ip6tables-nft to provide /usr/sbin/ip6tables (ip6tables) in manual mode
$ iptables --version
iptables v1.8.7 (nf_tables)
## docker version is 28.0.1
$ docker --version
Docker version 28.0.1, build 068a01e
## Cannot run docker with -p
$ docker run -dit -p 80:80 --rm --name alpine alpine:latest
c0371ee144a1dfb4b7d1da1c19c082b2576cfb2f6e5af47d199316fb3ce62243
docker: Error response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint alpine (a3cc07dee0bca76987b3d90104b3c58551f6e0ff0d8b18612d201b3c004347c2): Unable to enable DIRECT ACCESS FILTERING - DROP rule: (iptables failed: iptables --wait -t raw -A PREROUTING -p tcp -d 172.17.0.2 --dport 80 ! -i docker0 -j DROP: iptables v1.8.7 (nf_tables): unknown option "--dport"
Try `iptables -h' or 'iptables --help' for more information.
(exit status 2))
Run 'docker run --help' for more information
failed.
And, Ubuntu 20.04 on Raspberry Pi 4B
$ uname -srvp
Linux 5.4.0-1122-raspi #134-Ubuntu SMP PREEMPT Fri Jan 17 12:48:32 UTC 2025 aarch64
## pi4b(u20.04) has iptable_raw.ko
pi4b2:~$ find /lib/modules/$(uname -r) -name 'iptable_raw.ko'
/lib/modules/5.4.0-1122-raspi/kernel/net/ipv4/netfilter/iptable_raw.ko
pi4b2:~$ lsmod | grep -i iptable_raw
iptable_raw 16384 1
ip_tables 40960 11 iptable_filter,iptable_raw,iptable_nat
x_tables 57344 17 ip6table_filter,xt_conntrack,iptable_filter,xt_LOG,xt_tcpudp,xt_addrtype,xt_nat,ip6t_rt,xt_set,ip6_tables,ipt_REJECT,iptable_raw,ip_tables,xt_limit,xt_hl,xt_MASQUERADE,ip6t_REJECT
## tables are same as Jetson
pi4b2:~$ iptables --version
iptables v1.8.4 (legacy)
pi4b2:~$ arptables --version
arptables v1.8.4 (nf_tables)
pi4b2:~$ ebtables --version
ebtables 1.8.4 (nf_tables)
## docker version is 28.0.1
pi4b2:~$ docker --version
Docker version 28.0.1, build 068a01e
## running docker with -p
pi4b2:~$ docker run -dit -p 80:80 --rm --name alpine alpine:latest
3d38d290426ba3f5aa20781fc0c9a73c277a906b5dba8b42b108a11682fcbfd8
No problem with legacy iptables.
I re-build kernel with CONFIG_IP_NF_RAW=m and deployed iptable_raw.ko to my Jetson. In this case it works fine.
I’d be happy to check for you tomorrow on my home lab. I use portainer, so that specific error message o don’t think would be clear to me. Is there a way for me to check, or an article you could link me to to brush up my knowledge on debugging docker so I can better help?
Thanks @braymullo, much appreciated ... the error is the one that comes from docker run or docker compose up, as in earlier comments. But, I don't know where the error ends up with Portainer. No problem though, I think I have the info from @hiratagoh.
Thanks @hiratagoh ...
First, Jetson Linux(Ubuntu22.04) [...] docker: Error response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint alpine (a3cc07dee0bca76987b3d90104b3c58551f6e0ff0d8b18612d201b3c004347c2): Unable to enable DIRECT ACCESS FILTERING - DROP rule: (iptables failed: iptables --wait -t raw -A PREROUTING -p tcp -d 172.17.0.2 --dport 80 ! -i docker0 -j DROP: iptables v1.8.7 (nf_tables): unknown option "--dport"
The error is different unknown option "--dport", instead of can't initialize iptables table 'raw'. So, perhaps there's something else the Jetson kernel doesn't include, I'll take a look.
And, Ubuntu 20.04 on Raspberry Pi 4B
The kernel modules needed by these new iptables rules are available on a standard Ubuntu kernel so, that's the difference.
I am facing the same problem.. can’t initialize iptables table `raw’: Table does not exist (do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded.
on Jetson Orin Nano Super..
Hi @dilipbravo - if you can't add the missing kernel module, since release 28.0.2 you can set an environment variable to disable the raw rules ... but that does undo some of the security hardening added in the 28.0.0 release.
See the PR linked above (https://github.com/moby/moby/pull/49621), which includes an example of how to configure systemd if you're using it - and release notes https://github.com/moby/moby/releases/tag/v28.0.2.
