netavark
netavark copied to clipboard
containers
cannot start second container: iptables: Chain already exists
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug Description
I can start only one root container.
When I create or start the second root container I get this error message:
ERRO[0000] Starting some container dependencies
ERRO[0000] "netavark: code: 1, msg: iptables: Chain already exists.\n"
Error: error starting some containers: internal libpod error
I use nftables. When I restart nftables and all this extra stuff which podman puts into the firewall is gone I can create the second container.
My current /etc/containers/network/podman1.json looks like this:
{
"name": "podman1",
"id": "85ab0240733e50b6c3c1d497f5c882501bbb6d7476f66f5fa4cc9a4a0d11bc04",
"driver": "bridge",
"network_interface": "podman1",
"created": "2022-04-02T14:39:19.181441699+02:00",
"subnets": [
{
"subnet": "10.90.0.0/16",
"gateway": "10.90.0.1"
},
{
"subnet": "fdc2:4ba9:85d4:f3c1::/64",
"gateway": "fdc2:4ba9:85d4:f3c1::1"
},
{
"subnet": "2a02:xxx::/96"
}
],
"ipv6_enabled": true,
"internal": false,
"dns_enabled": true,
"ipam_options": {
"driver": "host-local"
}
}
Steps to reproduce the issue:
see above
Describe the results you received: I get the described error message from netavark regarding an already existing iptables chain.
Describe the results you expected: I expect to be able to start more than one root container.
Additional information you deem important (e.g. issue happens only occasionally):
Output of podman version:
Client: Podman Engine
Version: 4.0.2
API Version: 4.0.2
Go Version: go1.18beta2
Built: Thu Mar 3 15:56:09 2022
OS/Arch: linux/amd64
Output of podman info --debug:
host:
arch: amd64
buildahVersion: 1.24.1
cgroupControllers:
- cpuset
- cpu
- io
- memory
- hugetlb
- pids
- misc
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon-2.1.0-2.fc36.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.1.0, commit: '
cpus: 8
distribution:
distribution: fedora
variant: silverblue
version: "36"
eventLogger: journald
hostname: bat.localdomain
idMappings:
gidmap: null
uidmap: null
kernel: 5.17.1-300.fc36.x86_64
linkmode: dynamic
logDriver: journald
memFree: 3739787264
memTotal: 8211226624
networkBackend: netavark
ociRuntime:
name: crun
package: crun-1.4.4-1.fc36.x86_64
path: /usr/bin/crun
version: |-
crun version 1.4.4
commit: 6521fcc5806f20f6187eb933f9f45130c86da230
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
remoteSocket:
path: /run/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: true
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
version: |-
slirp4netns version 1.2.0-beta.0
commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
libslirp: 4.6.1
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.3
swapFree: 16800276480
swapTotal: 16800276480
uptime: 1h 16m 24.72s (Approximately 0.04 days)
plugins:
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
volume:
- local
registries:
search:
- registry.fedoraproject.org
- registry.access.redhat.com
- docker.io
- quay.io
store:
configFile: /etc/containers/storage.conf
containerStore:
number: 4
paused: 0
running: 4
stopped: 0
graphDriverName: overlay
graphOptions:
overlay.mountopt: nodev,metacopy=on
graphRoot: /var/mnt/data/containers/storage
graphStatus:
Backing Filesystem: xfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "true"
imageCopyTmpDir: /var/tmp
imageStore:
number: 40
runRoot: /var/run/containers/storage
volumePath: /var/mnt/data/containers/storage/volumes
version:
APIVersion: 4.0.2
Built: 1646319369
BuiltTime: Thu Mar 3 15:56:09 2022
GitCommit: ""
GoVersion: go1.18beta2
OsArch: linux/amd64
Version: 4.0.2
Package info (e.g. output of rpm -q podman or apt list podman):
podman-4.0.2-1.fc36.x86_64
Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)
Yes/No
Additional environment details (AWS, VirtualBox, physical, etc.):
Please run the command with --log-level debug and provide the output. I don't think we have any test for using more than one ipv6 subnet in the config, is there any reason you do this?
The reason why having an additional ULA might be useful is because the ::/64 network you get from your ISP might change.
I have removed one the ULA network. The error is the same.
{
"name": "podman1",
"id": "85ab0240733e50b6c3c1d497f5c882501bbb6d7476f66f5fa4cc9a4a0d11bc04",
"driver": "bridge",
"network_interface": "podman1",
"created": "2022-04-02T14:39:19.181441699+02:00",
"subnets": [
{
"subnet": "10.90.0.0/16",
"gateway": "10.90.0.1"
},
{
"subnet": "2a02:xxxx:ff::/96",
"gateway": "2a02:xxxx:ff:0:1"
}
],
"ipv6_enabled": true,
"internal": false,
"dns_enabled": true,
"ipam_options": {
"driver": "host-local"
}
}
Please find here the debug output:
I have tested again today. Always same result. The only way I can start both containers is by either stopping or restarting nftables.
For me this is a show stopper.
I have put this in my systemd service files which start the containers:
ExecStartPre=/usr/bin/systemctl reload nftables
ExecStartPost=/usr/bin/systemctl reload nftables
I consider this as a workaround. This faulty behaviour should be fixed.
Also it should be possible to disable firewall changes from podman.
As a basis I have this in my FORWARD and POSTROUTING.
podman works fine with this so I don't need any automatic changes.
table inet filter {
chain FORWARD {
type filter hook forward priority filter; policy drop;
oifname "podman0" ct state established,related accept
oifname "podman1" ct state established,related accept
iifname "podman0" oifname != "podman0" accept
iifname "podman1" oifname != "podman1" accept
}
}
table ip nat {
chain POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
oifname != "podman0" ip saddr 10.88.0.0/16 masquerade
oifname != "podman1" ip saddr 10.90.0.0/16 masquerade
}
}
I cannot reproduce this. Do you have ports forwarded for the containers?
Also can you check the netavark version /usr/libexec/podman/netavark version and the iptables version iptables --version.
I don't use port forwarding but my "inet filter" table (nftables) has a forward chain.
The root containers are using port 53 (bind) and 587 (postfix).
chain FORWARD {
type filter hook forward priority filter; policy drop;
oifname "podman0" ct state established,related accept
oifname "podman1" ct state established,related accept
oifname "virbr0" ct state established,related accept
oifname "podman1" ip daddr 10.90.0.2 tcp dport 53 accept
oifname "podman1" ip daddr 10.90.0.2 udp dport 53 accept
oifname "podman1" ip daddr 10.90.0.3 tcp dport 587 accept
oifname "podman1" tcp dport 389 accept
oifname "podman1" tcp dport 636 accept
oifname "podman1" tcp dport 5432 accept
iifname "virbr0" oifname != "virbr0" accept
iifname "podman0" oifname != "podman0" accept
iifname "podman1" oifname != "podman1" accept
}
Also I use masquerading in my "ip nat" table (postrouting chain):
chain POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
oifname != "podman0" ip saddr 10.88.0.0/16 masquerade
oifname != "podman1" ip saddr 10.90.0.0/16 masquerade
oifname != "virbr0" ip saddr 192.168.122.0/24 masquerade
}
Here the version output:
$ /usr/libexec/podman/netavark version
{
"version": "1.0.2",
"commit": "869f5cdebffc8495144a8601fd2a86dcb00d91d0",
"build_time": "2022-03-23T21:22:40.979038648+00:00",
"target": "x86_64-unknown-linux-gnu"
}
$ iptables --version
iptables v1.8.7 (nf_tables)
I am confused, so you are adding extra rules and then it starts failing? I think it is expected that we can conflict if there are conflicting rules already present.
I don't understand what you mean. My nftables is configured exactly as before netavark was introduced.
There would not be any conflict if netavark would not try to add rules.
If you manually configure conflicting nftables rules I see no reason why netavark has to work in this case.
If you do not manually set the rules then I do not understand what is needed to reproduce this isuues. Because for me your config from the issue works fine.
To me it looks like as if netavark does not expect the user to work with a static firewall.
This is related to this:
https://github.com/containers/netavark/issues/339
For me it was a bug because the issue occured after updating to Fedora 36. With podman 3.x my containers where available after the start of my PC - after the update always one container was down because of this issue.
If you think this is not a bug then it would be good to implement this config flag that the user has the possibility to disable firewall changes by podman or netavark respectively.
I have this issue as well. Always one container. The fix mentined by @fansari does not seem to work on Fedora CoreOS because nftables does not seem to be active.
I use podman and nftables on Fedora 36 Sliverblue and Fedora 36 CoreOS. On both systems I run nftables.
When I build a new container I always have to stop nftables right in front - otherwise I will get this error because of conflicting rule.
Same thing happens when I start my containers.
As I workaround I have written some python script which adds lines for ExecStartPre and ExecStartPost into the service files generated by "podman generate system" I use to startup my containers. This way every podman service file will reloads nftables before and after a container is started.
Hopefully the developers will soon bring some option which allows the user to use podman containers without touching the existing firewall rules.
Hi all,
I have encountered the same problem on RHEL 8.6. Any solutions for that?
Have the same issue on Debian Bookworm.
Its weird, as some custom rules have the sideeffect of empty chains remaining. Log is attached. Debian iptables is a wrapper over the nftables tools, which have some sublte differences.
The log shows the commands and nftables dumps between them.
Well, that might be the issue:
# iptables -t filter -S
iptables v1.8.8 (nf_tables): table `filter' is incompatible, use 'nft' tool.
I still use my workaround. I could not find any other way to disable the rules which netavark brings in.
A good solution would be like in docker where the user can disable any firewall changes by docker completely.
I'm a little late but this doesn't seem to be referenced anywhere else, and is still a problem apparently since I was just getting the same error after moving my containers to a NixOS machine that also acts as a router & firewall.
It appears that netavark relies on iptables-nft on machines running nftables, and is attempting to add chains that may already exist on the host, such as PREROUTING, POSTROUTING, FORWARD, etc.
Anyway, I found that renaming my tables appears to resolve it, e.g.:
table ip filter {... becomes table ip host-filter {...
table ip nat {... becomes table ip host-nat {...
Now after I start my containers, I can run nftables list ruleset and see the netavark rules below mine in their own separate tables and everything seems to be working.
